ibeis.algo.preproc package¶

Submodules¶

ibeis.algo.preproc.occurrence_blackbox module¶

ibeis.algo.preproc.occurrence_blackbox.cluster_timespace(X_data, thresh, km_per_sec=0.02)[source]¶

Agglometerative clustering of time/space data

Parameters:	X_data (ndarray) – Nx3 array where columns are (seconds, lat, lon) thresh (float) – threshold in kilometers

References

http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/

scipy.cluster.hierarchy.linkage.html scipy.cluster.hierarchy.fcluster.html

Notes

# Visualize spots http://www.darrinward.com/lat-long/?id=2009879

Example

>>> # DISABLE_DOCTEST
>>> from occurrence_blackbox import *  # NOQA
>>> # Nx1 matrix denoting groundtruth locations (for testing)
>>> X_name = np.array([0, 1, 1, 1, 1, 1, 2, 2, 2])
>>> # Nx3 matrix where each columns are (time, lat, lon)
>>> X_data = np.array([
>>>     (0, 42.727985, -73.683994),  # MRC
>>>     (0, 42.657414, -73.774448),  # Park1
>>>     (0, 42.658333, -73.770993),  # Park2
>>>     (0, 42.654384, -73.768919),  # Park3
>>>     (0, 42.655039, -73.769048),  # Park4
>>>     (0, 42.657872, -73.764148),  # Park5
>>>     (0, 42.876974, -73.819311),  # CP1
>>>     (0, 42.862946, -73.804977),  # CP2
>>>     (0, 42.849809, -73.758486),  # CP3
>>> ])
>>> thresh = 5.0  # kilometers
>>> X_labels = cluster_timespace(X_data, thresh)
>>> result = ('X_labels = %r' % (X_labels,))
>>> print(result)
X_labels = array([3, 2, 2, 2, 2, 2, 1, 1, 1], dtype=int32)

ibeis.algo.preproc.occurrence_blackbox.haversine(latlon1, latlon2)[source]¶

Calculate the great circle distance between two points on the earth (specified in decimal degrees)

Parameters:	latlon1 (tuple) – (lat, lon) latlon2 (tuple) – (lat, lon)
Returns:	distance in kilometers
Return type:	float

References

en.wikipedia.org/wiki/Haversine_formula gis.stackexchange.com/questions/81551/matching-gps-tracks stackoverflow.com/questions/4913349/haversine-distance-gps-points

Example

>>> from occurrence_blackbox import *  # NOQA
>>> import scipy.spatial.distance as spdist
>>> import functools
>>> latlon1 = [-80.21895315, -158.81099213]
>>> latlon2 = [  9.77816711,  -17.27471498]
>>> kilometers = haversine(latlon1, latlon2)
>>> result = ('kilometers = %s' % (kilometers,))
>>> print(result)
kilometers = 11930.9093642

ibeis.algo.preproc.occurrence_blackbox.timespace_distance(pt1, pt2, km_per_sec=0.02)[source]¶

Computes distance between two points in space and time. Time is converted into spatial units using km_per_sec

Parameters:	pt1 (tuple) – (seconds, lat, lon) pt2 (tuple) – (seconds, lat, lon) km_per_sec (float) – reasonable animal walking speed
Returns:	distance in kilometers
Return type:	float

Example

>>> from occurrence_blackbox import *  # NOQA
>>> import scipy.spatial.distance as spdist
>>> import functools
>>> km_per_sec = .02
>>> latlon1 = [-80.21895315, -158.81099213]
>>> latlon2 = [  9.77816711,  -17.27471498]
>>> pt1 = [360.0] + latlon1
>>> pt2 = [0.0] + latlon2
>>> kilometers = timespace_distance(pt1, pt2)
>>> result = ('kilometers = %s' % (kilometers,))
>>> print(result)
kilometers = 2058.6323187

ibeis.algo.preproc.preproc_annot module¶

The goal of this module is to offload annotation work from the controller into a single place.

CommandLine Help for manual controller functions

# Cross platform alias helper python -c “import utool as ut; ut.write_to(‘Tgen.sh’, ‘python -m ibeis.control.template_generator $@’)” # NOQA

Tgen.sh –tbls annotations –Tcfg with_getters:True strip_docstr:False with_columns:False Tgen.sh –tbls annotations –Tcfg with_getters:True with_native:True strip_docstr:True Tgen.sh –tbls annotations –Tcfg with_getters:True strip_docstr:True with_columns:False –quiet Tgen.sh –tbls annotations –Tcfg with_getters:True strip_docstr:False with_columns:False

ibeis.algo.preproc.preproc_annot.generate_annot_properties(ibs, gid_list, bbox_list=None, theta_list=None, species_list=None, nid_list=None, name_list=None, detect_confidence_list=None, notes_list=None, vert_list=None, annot_uuid_list=None, yaw_list=None, quiet_delete_thumbs=False)[source]¶

ibeis.algo.preproc.preproc_annot.make_annot_semantic_uuid(semantic_infotup)[source]¶

Parameters:	semantic_infotup (tuple) – (image_uuid_list, verts_list, theta_list, view_list, name_list, species_list)
Returns:	annot_semantic_uuid_list
Return type:	list

CommandLine:

python -m ibeis.algo.preproc.preproc_annot --test-make_annot_semantic_uuid

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_annot import *  # NOQA
>>> ibs, aid_list = testdata_preproc_annot()
>>> semantic_infotup = ibs.get_annot_semantic_uuid_info(aid_list)
>>> annot_semantic_uuid_list = make_annot_semantic_uuid(semantic_infotup)
>>> result = str(annot_semantic_uuid_list[0])
>>> print(result)
bf774bf3-582d-dbce-6ca6-329adeb086a6

215ab5f9-fe53-d7d1-59b8-d6b5ce7e6ca6

ibeis.algo.preproc.preproc_annot.make_annot_visual_uuid(visual_infotup)[source]¶

Parameters:	visual_infotup (tuple) – (image_uuid_list, verts_list, theta_list)
Returns:	annot_visual_uuid_list
Return type:	list

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_annot import *  # NOQA
>>> ibs, aid_list = testdata_preproc_annot()
>>> visual_infotup = ibs.get_annot_visual_uuid_info(aid_list)
>>> annot_visual_uuid_list = make_annot_visual_uuid(visual_infotup)
>>> result = str(annot_visual_uuid_list[0])
>>> print(result)
8687dcb6-1f1f-fdd3-8b72-8f36f9f41905

ibeis.algo.preproc.preproc_annot.make_annotation_uuids(image_uuid_list, bbox_list, theta_list, deterministic=True)[source]¶

ibeis.algo.preproc.preproc_annot.postget_annot_verts(vertstr_list)[source]¶

ibeis.algo.preproc.preproc_annot.test_annotation_uuid(ibs)[source]¶

Consistency test

ibeis.algo.preproc.preproc_annot.testdata_preproc_annot()[source]¶

ibeis.algo.preproc.preproc_image module¶

ibeis.algo.preproc.preproc_image.add_images_params_gen(gpath_list, **kwargs)[source]¶

generates values for add_images sqlcommands asychronously

Parameters:	gpath_list (list) –

Kwargs:

ordered, force_serial, chunksize, prog, verbose, quiet, nTasks, freq, adjust

Returns:	params_gen
Return type:	generator

CommandLine:

python -m ibeis.algo.preproc.preproc_image --exec-add_images_params_gen

Example0:

>>> # ENABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_image import *   # NOQA
>>> from vtool.tests import grabdata
>>> gpath_list = grabdata.get_test_gpaths(ndata=3) + ['doesnotexist.jpg']
>>> params_list = list(add_images_params_gen(gpath_list))
>>> assert str(params_list[0][0]) == '66ec193a-1619-b3b6-216d-1784b4833b61', 'UUID gen method changed'
>>> assert str(params_list[0][3]) == 'easy1.JPG', 'orig name is different'
>>> assert params_list[3] is None

ibeis.algo.preproc.preproc_image.get_standard_ext(gpath)[source]¶

Returns standardized image extension

Cyth::

cdef:

str gpath str ext

ibeis.algo.preproc.preproc_image.on_delete(ibs, featweight_rowid_list, qreq_=None)[source]¶

ibeis.algo.preproc.preproc_image.parse_exif(pil_img)[source]¶

Image EXIF helper

Cyth::

cdef:

Image pil_img dict exif_dict long lat long lon long exiftime

ibeis.algo.preproc.preproc_image.parse_imageinfo(gpath)[source]¶

Worker function: gpath must be in UNIX-PATH format!

Parameters:	tup (tuple) – a tuple or one argument (so the function can be parallelized easily) (here it is just gpath, no tuple, sorry for confusion)
Returns:	param_tup - if successful returns a tuple of image parameters which are values for SQL columns on else returns None
Return type:	tuple

CommandLine:

python -m ibeis.algo.preproc.preproc_image --exec-parse_imageinfo

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_image import *  # NOQA
>>> gpath = ('/media/raid/work/lynx/_ibsdb/images/f6c84c6d-55ca-fd02-d0b4-1c7c9c27c894.jpg')
>>> param_tup = parse_imageinfo(tup)
>>> result = ('param_tup = %s' % (str(param_tup),))
>>> print(result)

ibeis.algo.preproc.preproc_occurrence module¶

ibeis.algo.preproc.preproc_occurrence.agglomerative_cluster_occurrences(X_data, seconds_thresh)[source]¶

Agglomerative occurrence clustering algorithm

Parameters:	X_data (ndarray) – Length N array of data to cluster seconds_thresh (float) –
Returns:	(label_arr) - Length N array of cluster indexes
Return type:	ndarray

CommandLine:

python -m ibeis.algo.preproc.preproc_occurrence --exec-agglomerative_cluster_occurrences

References

https://docs.scipy.org/doc/scipy-0.9.0/reference/generated/scipy.cluster.hierarchy.fclusterdata.html#scipy.cluster.hierarchy.fclusterdata http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.cluster.hierarchy.fcluster.html

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_occurrence import *  # NOQA
>>> X_data = '?'
>>> seconds_thresh = '?'
>>> (occur_ids, occur_gids) = agglomerative_cluster_occurrences(X_data, seconds_thresh)
>>> result = ('(occur_ids, occur_gids) = %s' % (str((occur_ids, occur_gids)),))
>>> print(result)

ibeis.algo.preproc.preproc_occurrence.cluster_timespace(X_data, thresh)[source]¶

References

http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/

scipy.cluster.hierarchy.linkage.html

CommandLine:

python -m ibeis.algo.preproc.preproc_occurrence cluster_timespace --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_occurrence import *  # NOQA
>>> X_data = testdata_gps()
>>> thresh = 10
>>> X_labels = cluster_timespace(X_data, thresh)
>>> fnum = pt.ensure_fnum(None)
>>> fig = pt.figure(fnum=fnum, doclf=True, docla=True)
>>> hier.dendrogram(linkage_mat, orientation='top')
>>> plot_annotaiton_gps(X_data)
>>> ut.show_if_requested()

ibeis.algo.preproc.preproc_occurrence.compute_occurrence_groups(ibs, gid_list, cluster_algo, cfgdict={}, use_gps=False, verbose=None)[source]¶

Parameters:	ibs (IBEISController) – ibeis controller object gid_list (list) –
Returns:	(None, None)
Return type:	tuple

CommandLine:

python -m ibeis --tf compute_occurrence_groups
python -m ibeis --tf compute_occurrence_groups --show --zoom=.3

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_occurrence import *  # NOQA
>>> import ibeis
>>> import vtool as vt
>>> #ibs = ibeis.opendb(defaultdb='testdb1')
>>> ibs = ibeis.opendb(defaultdb='PZ_Master1')
>>> gid_list = ibs.get_valid_gids(require_unixtime=True, require_gps=True)
>>> use_gps = True
>>> cluster_algo = 'meanshift'
>>> cfgdict = dict(quantile=.005, min_imgs_per_occurence=2)
>>> (occur_labels, occur_gids) = compute_occurrence_groups(ibs, gid_list, cluster_algo, cfgdict, use_gps=use_gps)
>>> aidsgroups_list = ibs.unflat_map(ibs.get_image_aids, occur_gids)
>>> aids_list = list(map(ut.flatten, aidsgroups_list))
>>> nids_list = list(map(np.array, ibs.unflat_map(ibs.get_annot_name_rowids, aids_list)))
>>> metric = [len(np.unique(nids[nids > -1])) for nids in nids_list]
>>> metric = [vt.safe_max(np.array(ut.dict_hist(nids).values())) for nids in nids_list]
>>> #metric = list(map(len, aids_list))
>>> sortx = ut.list_argsort(metric)[::-1]
>>> index = sortx[20]
>>> #gids = occur_gids[index]
>>> aids = aids_list[index]
>>> aids = ibs.filter_annots_general(aids, min_qual='ok', is_known=True)
>>> gids = list(set(ibs.get_annot_gids(aids)))
>>> print('len(aids) = %r' % (len(aids),))
>>> img_list = ibs.get_images(gids)
>>> ut.quit_if_noshow()
>>> from ibeis.viz import viz_graph
>>> import plottool as pt
>>> #pt.imshow(bigimg)
>>> #aids = ibs.group_annots_by_name(aids)[0][0]
>>> self = viz_graph.make_name_graph_interaction(ibs, aids=aids,
>>>                                              with_all=False,
>>>                                              prog='neato')
>>> ut.show_if_requested()

ibs.unflat_map(ibs.get_annot_case_tags, aids_list) ibs.filter_aidpairs_by_tags(has_any=’photobomb’)

photobomb_aids = ibs.filter_aidpairs_by_tags(has_any=’photobomb’) aids = photobomb_aids[0:10].flatten() _gt_aids = ibs.get_annot_groundtruth(aids) gt_aids = ut.get_list_column_slice(_gt_aids, slice(0, 3)) aid_set = np.unique(np.append(aids.flatten(), ut.flatten(gt_aids))) aid_set = ibs.filter_annots_general(aid_set, minqual=’ok’)

# This is the set of annotations used for testing intraoccurrence photobombs #print(ut.repr3(ibeis.other.dbinfo.get_dbinfo(ibs, aid_list=aid_set), strvals=True, nl=1)) print(ut.repr3(ibs.get_annot_stats_dict(aid_set, forceall=True), strvals=True, nl=1))

ibeis.algo.preproc.preproc_occurrence.compute_occurrence_unixtime(ibs, occur_gids)[source]¶

ibeis.algo.preproc.preproc_occurrence.filter_and_relabel(labels, label_gids, min_imgs_per_occurence, occur_unixtimes=None)[source]¶

Removes clusters with too few members. Relabels clusters-labels such that label 0 has the most members

ibeis.algo.preproc.preproc_occurrence.group_images_by_label(label_arr, gid_arr)[source]¶

Input: Length N list of labels and ids Output: Length M list of unique labels, and lenth M list of lists of ids

ibeis.algo.preproc.preproc_occurrence.ibeis_compute_occurrences(ibs, gid_list, seconds_thresh=None, verbose=None)[source]¶

clusters occurrences togethers (by time, not yet space) An occurrence is a meeting, localized in time and space between a camera and a group of animals. Animals are identified within each occurrence.

Does not modify database state, just returns cluster ids

Parameters:	ibs (IBEISController) – ibeis controller object gid_list (list) –
Returns:	(None, None)
Return type:	tuple

CommandLine:

python -m ibeis --tf ibeis_compute_occurrences:0 --show

TODO: FIXME: good example of autogen doctest return failure

#Ignore: # >>> import ibeis # >>> from ibeis.algo.preproc.preproc_occurrence import * # NOQA # >>> ibs = ibeis.opendb(defaultdb=’lynx’) # >>> aid_list = ibs.get_valid_aids() # >>> filter_kw = {} # >>> filter_kw[‘been_adjusted’] = True # >>> aid_list_ = ibs.filter_annots_general(aid_list, filter_kw) # >>> gid_list = ibs.get_annot_gids(aid_list_) # >>> flat_imgsetids, flat_gids = ibeis_compute_occurrences(ibs, gid_list) # >>> aids_list = list(ut.group_items(aid_list_, flat_imgsetids).values()) # >>> metric = list(map(len, aids_list)) # >>> sortx = ut.list_argsort(metric)[::-1] # >>> index = sortx[1] # >>> #gids = occur_gids[index] # >>> aids = aids_list[index] # >>> gids = list(set(ibs.get_annot_gids(aids))) # >>> print(‘len(aids) = %r’ % (len(aids),)) # >>> ut.quit_if_noshow() # >>> from ibeis.viz import viz_graph # >>> import plottool as pt # >>> #pt.imshow(bigimg) # >>> #bigimg = vt.stack_image_recurse(img_list) # >>> self = viz_graph.make_name_graph_interaction(ibs, aids=aids, with_all=False) # >>> ut.show_if_requested()

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_occurrence import *  # NOQA
>>> import ibeis
>>> ibs = ibeis.opendb(defaultdb='testdb1')
>>> gid_list = ibs.get_valid_gids()
>>> aid_list_ = ibs.filter_annots_general(aid_list, filter_kw)
>>> (flat_imgsetids, flat_gids) = ibeis_compute_occurrences(ibs, gid_list)
>>> aids_list = list(ut.group_items(aid_list_, flat_imgsetids).values())
>>> metric = list(map(len, aids_list))
>>> sortx = ut.list_argsort(metric)[::-1]
>>> index = sortx[1]
>>> #gids = occur_gids[index]
>>> aids = aids_list[index]
>>> gids = list(set(ibs.get_annot_gids(aids)))
>>> print('len(aids) = %r' % (len(aids),))
>>> print(result)
>>> ut.quit_if_noshow()
>>> from ibeis.viz import viz_graph
>>> import plottool as pt
>>> #pt.imshow(bigimg)
>>> #bigimg = vt.stack_image_recurse(img_list)
>>> self = viz_graph.make_name_graph_interaction(ibs, aids=aids,
>>>                                              with_all=False,
>>>                                              prog='neato')
>>> ut.show_if_requested()

ibeis.algo.preproc.preproc_occurrence.meanshift_cluster_occurrences(X_data, quantile)[source]¶

Meanshift occurrence clustering algorithm

Parameters:	X_data (ndarray) – Length N array of data to cluster quantile (float) – quantile should be between [0, 1]. eg: quantile=.5 represents the median of all pairwise distances
Returns:	Length N array of labels
Return type:	ndarray

CommandLine:

python -m ibeis.algo.preproc.preproc_occurrence --exec-meanshift_cluster_occurrences

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.algo.preproc.preproc_occurrence import *  # NOQA
>>> X_data = '?'
>>> quantile = '?'
>>> result = meanshift_cluster_occurrences(X_data, quantile)
>>> print(result)

ibeis.algo.preproc.preproc_occurrence.plot_annotaiton_gps(X_data)[source]¶

Plots gps coordinates on a map projection

InstallBasemap:

sudo apt-get install libgeos-dev pip install git+https://github.com/matplotlib/basemap

ibeis.algo.preproc.preproc_occurrence.prepare_X_data(ibs, gid_list, use_gps=False)[source]¶

FIXME: use vt.haversine formula on gps dimensions fix weighting between seconds and gps

ibeis.algo.preproc.preproc_occurrence.testdata_gps()[source]¶

Simple data to test GPS algorithm.

Returns:	X_name – Nx1 matrix denoting groundtruth locations X_data (ndarray): Nx3 matrix where each columns are (time, lat, lon)
Return type:	ndarray

ibeis.algo.preproc.preproc_occurrence.timespace_distance(pt1, pt2)[source]¶

ibeis.algo.preproc.preproc_occurrence.timespace_pdist(X_data)[source]¶

ibeis.algo.preproc.preproc_residual module¶

ibeis.algo.preproc.preproc_residual.add_residual_params_gen(ibs, fid_list, qreq_=None)[source]¶

ibeis.algo.preproc.preproc_residual.on_delete(ibs, featweight_rowid_list)[source]¶

ibeis.algo.preproc.preproc_rvec module¶

ibeis.algo.preproc.preproc_rvec.add_rvecs_params_gen(ibs, nInput=None)[source]¶

ibeis.algo.preproc.preproc_rvec.generate_rvecs(vecs_list, words)[source]¶

Module contents¶

ibeis.algo.preproc.reassign_submodule_attributes(verbose=True)[source]¶

why reloading all the modules doesnt do this I don’t know

ibeis.algo.preproc.reload_subs(verbose=True)[source]¶

Reloads ibeis.algo.preproc and submodules

ibeis.algo.preproc.rrrr(verbose=True)¶

Reloads ibeis.algo.preproc and submodules