utool package

Submodules

utool.DynamicStruct module

class utool.DynamicStruct.DynStruct(child_exclude_list=[], copy_dict=None, copy_class=None)[source]

Bases: utool.Printable.AbstractPrintable

dynamically add and remove members

add_dict(dyn_dict)[source]

Adds a dictionary to the prefs

copy()[source]
deepcopy(**kwargs)[source]
execstr(local_name)[source]

returns a string which when evaluated will add the stored variables to the current namespace

localname is the name of the variable in the current scope * use locals().update(dyn.to_dict()) instead

flat_dict(dyn_dict={}, only_public=True)[source]
to_dict()[source]

Converts dynstruct to a dictionary.

update(**kwargs)[source]

utool.Preferences module

FIXME:
This class is very old, convoluted, and coupled. It really needs to be rewritten efficiently. the __setattr__ __getattr__ stuff needs to be redone, and DynStruct probably needs to go away.
class utool.Preferences.Pref(default=<class 'utool.DynamicStruct.DynStruct'>, doc='empty docs', hidden=False, choices=None, depeq=None, fpath='', name='root', parent=None)[source]

Bases: utool.DynamicStruct.DynStruct

Structure for Creating Preferences. Caveats: When using a value call with [‘valname’] to be safe Features: * Can be saved and loaded. * Can be nested * Dynamically add/remove

__getattr__(name)[source]

Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception. Get a child from this parent called as last resort. Allows easy access to internal prefs

__setattr__(name, attr)[source]

Called when an attribute assignment is attempted. This is called instead of the normal mechanism (i.e. store the value in the instance dictionary). name is the attribute name, value is the value to be assigned to it.

If __setattr__() wants to assign to an instance attribute, it should not simply execute self.name = value this would cause a recursive call to itself. Instead, it should insert the value in the dictionary of instance attributes, e.g., self.__dict__[name] = value. For new-style classes, rather than accessing the instance dictionary, it should call the base class method with the same name, for example, object.__setattr__(self, name, value). ‘Wraps child attributes in a Pref object if not already done’

change_combo_val(new_val)[source]

Checks to see if a selection is a valid index or choice of a combo preference

createQWidget()[source]
customPrintableType(name)[source]
ensure_attr(attr, default)[source]
full_name()[source]

returns name all the way up the tree

get_fpath()[source]
get_printable(type_bit=True, print_exclude_aug=[])[source]
get_type()[source]
items()[source]

Wow this class is messed up. I had to overwrite items when moving to python3, just because I haden’t called it yet

iteritems()[source]

Wow this class is messed up. I had to overwrite items when moving to python3, just because I haden’t called it yet

load()[source]

Read pref dict stored on disk. Overwriting current values.

pref_update(key, new_val)[source]

Changes a preference value and saves it to disk

qt_col_count()[source]
qt_get_child(row)[source]
qt_get_data(column)[source]
qt_get_parent()[source]
qt_is_editable()[source]
qt_parents_index_of_me()[source]
qt_row_count()[source]
qt_set_leaf_data(qvar)[source]
save()[source]

Saves prefs to disk in dict format

to_dict(split_structs_bit=False)[source]

Converts prefeters to a dictionary. Children Pref can be optionally separated

toggle(key)[source]

Toggles a boolean key

update(**kwargs)[source]
value()[source]
class utool.Preferences.PrefChoice(choices, default)[source]

Bases: utool.DynamicStruct.DynStruct

change_val(new_val)[source]
combo_val()[source]
get_tuple()[source]
class utool.Preferences.PrefInternal(_intern, name, doc, default, hidden, fpath, depeq, choices)[source]

Bases: utool.DynamicStruct.DynStruct

freeze_type(_intern)[source]
get_type(_intern)[source]
class utool.Preferences.PrefTree(_tree, parent)[source]

Bases: utool.DynamicStruct.DynStruct

utool.Preferences.test_Preferences()[source]

CommandLine:

python -m utool.Preferences --test-test_Preferences --show --verbpref

Example

>>> # DISABLE_DOCTEST
>>> # FIXME depends on guitool
>>> from utool.Preferences import *  # NOQA
>>> import utool as ut
>>> import guitool
>>> guitool.ensure_qtapp()
>>> root = test_Preferences()
>>> ut.quit_if_noshow()
>>> widget = root.createQWidget()
>>> #widget.show()
>>> guitool.qtapp_loop(widget)

utool.Printable module

class utool.Printable.AbstractPrintable(child_print_exclude=[])[source]

Bases: object

A base class that prints its attributes instead of the memory address

format_printable(type_bit=False, indstr=' * ')[source]
get_printable(type_bit=True, print_exclude_aug=[], val_bit=True, max_valstr=-1, justlength=False)[source]
printme()[source]
printme2(type_bit=True, print_exclude_aug=[], val_bit=True, max_valstr=-1, justlength=True)[source]
printme3()[source]

utool.__main__ module

utool.__main__.utool_main()[source]

utool.sandbox module

class utool.sandbox.LazyModule(modname)[source]

Bases: object

Waits to import the module until it is actually used

__getattr__(item)[source]

Maps values to attributes. Only called if there isn’t an attribute with this name

__setattr__(item, value)[source]

Maps attributes to values. Only if we are initialised

utool.util_alg module

utool.util_alg.absdiff(x, y)[source]
utool.util_alg.almost_allsame(vals)[source]
utool.util_alg.almost_eq(arr1, arr2, thresh=1e-11, ret_error=False)[source]

checks if floating point number are equal to a threshold

utool.util_alg.apply_grouping(items, groupxs)[source]

applies grouping from group_indicies non-optimized version

Parameters:
  • items (list) – items to group
  • groupxs (list of list of ints) – grouped lists of indicies
SeeAlso:
vt.apply_grouping - optimized numpy version ut.group_indices

CommandLine:

python -m utool.util_alg --exec-apply_grouping --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> idx2_groupid = [2, 1, 2, 1, 2, 1, 2, 3, 3, 3, 3]
>>> items        = [1, 8, 5, 5, 8, 6, 7, 5, 3, 0, 9]
>>> (keys, groupxs) = ut.group_indices(idx2_groupid)
>>> grouped_items = ut.apply_grouping(items, groupxs)
>>> result = ut.repr2(grouped_items)
>>> print(result)
[[8, 5, 6], [1, 5, 8, 7], [5, 3, 0, 9]]
utool.util_alg.bayes_rule(b_given_a, prob_a, prob_b)[source]

P(A | B) = frac{ P(B | A) P(A) }{ P(B) }

Parameters:
  • b_given_a (ndarray or float) –
  • prob_a (ndarray or float) –
  • prob_b (ndarray or float) –
Returns:

a_given_b
Return type:

ndarray or float

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> b_given_a = .1
>>> prob_a = .3
>>> prob_b = .4
>>> a_given_b = bayes_rule(b_given_a, prob_a, prob_b)
>>> result = a_given_b
>>> print(result)
0.075
utool.util_alg.choose(n, k)[source]

N choose k

binomial combination (without replacement) scipy.special.binom

utool.util_alg.colwise_diag_idxs(size, num=2)[source]

dont trust this implementation or this function name

Parameters:size (int) –
Returns:upper_diag_idxs
Return type:?

CommandLine:

python -m utool.util_alg --exec-colwise_diag_idxs --size=5 --num=2
python -m utool.util_alg --exec-colwise_diag_idxs --size=3 --num=3

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> size = ut.get_argval('--size', default=5)
>>> num = ut.get_argval('--num', default=2)
>>> mat = np.zeros([size] * num, dtype=np.int)
>>> upper_diag_idxs = colwise_diag_idxs(size, num)
>>> poses = np.array(upper_diag_idxs)
>>> idxs = np.ravel_multi_index(poses.T, mat.shape)
>>> print('poses.T =\n%s' % (ut.repr2(poses.T),))
>>> mat[tuple(poses.T)] = np.arange(1, len(poses) + 1)
>>> print(mat)
poses.T =
np.array([[0, 0, 1, 0, 1, 2, 0, 1, 2, 3],
          [1, 2, 2, 3, 3, 3, 4, 4, 4, 4]])
utool.util_alg.compare_groupings(groups1, groups2)[source]

Returns a measure of how disimilar two groupings are

Parameters:
  • groups1 (list) – grouping of items
  • groups2 (list) – grouping of items

CommandLine:

python -m utool.util_alg --exec-compare_groupings
SeeAlso:
vtool.group_indicies vtool.apply_grouping
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> groups1 = [[1, 2, 3], [4], [5, 6], [7, 8], [9, 10, 11]]
>>> groups2 = [[1, 2, 11], [3, 4], [5, 6], [7], [8, 9], [10]]
>>> total_error = compare_groupings(groups1, groups2)
>>> result = ('total_error = %r' % (total_error,))
>>> print(result)
total_error = 20
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> groups1 = [[1, 2, 3], [4], [5, 6]]
>>> groups2 = [[1, 2, 3], [4], [5, 6]]
>>> total_error = compare_groupings(groups1, groups2)
>>> result = ('total_error = %r' % (total_error,))
>>> print(result)
total_error = 0
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> groups1 = [[1, 2, 3], [4], [5, 6]]
>>> groups2 = [[1, 2], [4], [5, 6]]
>>> total_error = compare_groupings(groups1, groups2)
>>> result = ('total_error = %r' % (total_error,))
>>> print(result)
total_error = 4
utool.util_alg.cumsum(num_list)[source]

python cumsum

References

stackoverflow.com/questions/9258602/elegant-pythonic-cumsum

utool.util_alg.deg_to_rad(degree)[source]
utool.util_alg.diagonalized_iter(size)[source]

TODO: generalize to more than 2 dimensions to be more like itertools.product.

CommandLine:

python -m utool.util_alg --exec-diagonalized_iter
python -m utool.util_alg --exec-diagonalized_iter --size=5

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> size = ut.get_argval('--size', default=4)
>>> iter_ = diagonalized_iter(size)
>>> mat = [[None] * size for _ in range(size)]
>>> for count, (r, c) in enumerate(iter_):
>>>     mat[r][c] = count
>>> result = ut.repr2(mat, nl=1, packed=True)
>>> print(result)
[[0, 2, 5, 9],
 [1, 4, 8, 12],
 [3, 7, 11, 14],
 [6, 10, 13, 15],]
utool.util_alg.edit_distance(string1, string2)[source]
utool.util_alg.enumerate_primes(max_prime=4100)[source]
utool.util_alg.euclidean_dist(vecs1, vec2, dtype=None)[source]
utool.util_alg.fibonacci(n)
Parameters:n (int) –
Returns:a
Return type:int

References

http://stackoverflow.com/questions/15047116/a-iterative-algorithm-for-fibonacci-numbers

CommandLine:

python -m utool.util_alg --test-fibonacci_iterative

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> with ut.Timer('fib iter'):
>>>     series = [fibonacci_iterative(n) for n in range(20)]
>>> result = ('series = %s' % (str(series[0:10]),))
>>> print(result)
series = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
utool.util_alg.fibonacci_iterative(n)[source]
Parameters:n (int) –
Returns:a
Return type:int

References

http://stackoverflow.com/questions/15047116/a-iterative-algorithm-for-fibonacci-numbers

CommandLine:

python -m utool.util_alg --test-fibonacci_iterative

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> with ut.Timer('fib iter'):
>>>     series = [fibonacci_iterative(n) for n in range(20)]
>>> result = ('series = %s' % (str(series[0:10]),))
>>> print(result)
series = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
utool.util_alg.fibonacci_recursive(n)[source]

CommandLine:

python -m utool.util_alg --test-fibonacci_recursive

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> with ut.Timer('fib rec'):
>>>     series = [fibonacci_recursive(n) for n in range(20)]
>>> result = ('series = %s' % (str(series[0:10]),))
>>> print(result)
series = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
utool.util_alg.find_grouping_consistencies(groups1, groups2)[source]

Returns a measure of group consistency

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> groups1 = [[1, 2, 3], [4], [5, 6]]
>>> groups2 = [[1, 2], [4], [5, 6]]
>>> common_groups = find_grouping_consistencies(groups1, groups2)
>>> result = ('common_groups = %r' % (common_groups,))
>>> print(result)
common_groups = [(5, 6), (4,)]
utool.util_alg.flatten_membership_mapping(uid_list, members_list)[source]
utool.util_alg.generate_primes(start_guess=2)[source]
utool.util_alg.get_nth_bell_number(n)[source]

Returns the (num_items - 1)-th Bell number using recursion. The Bell numbers count the number of partitions of a set.

Parameters:n (int) – number of items in a set
Returns:
Return type:int

References

http://adorio-research.org/wordpress/?p=11460

CommandLine:

python -m utool.util_alg --exec-bell --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> n = 3
>>> result = get_nth_bell_number(n)
>>> print(result)
5
utool.util_alg.get_nth_prime(n, max_prime=4100, safe=True)[source]

hacky but still brute force algorithm for finding nth prime for small tests

utool.util_alg.get_nth_prime_bruteforce(n, start_guess=2, start_num_primes=0)[source]
utool.util_alg.get_phi()[source]

Golden Ratio: phi = 1 / sqrt(5) / 2.0 = 1.61803398875

utool.util_alg.get_phi_ratio1()[source]
utool.util_alg.get_prime_index(prime)[source]
utool.util_alg.greedy_max_inden_setcover(candidate_sets_dict, items, max_covers=None)[source]

greedy algorithm for maximum independent set cover

Covers items with sets from candidate sets. Could be made faster.

CommandLine:

python -m utool.util_alg --test-greedy_max_inden_setcover
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> candidate_sets_dict = {'a': [5, 3], 'b': [2, 3, 5],
...                        'c': [4, 8], 'd': [7, 6, 2, 1]}
>>> items = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> max_covers = None
>>> tup = greedy_max_inden_setcover(candidate_sets_dict, items, max_covers)
>>> (uncovered_items, covered_items_list, accepted_keys) = tup
>>> result = ut.list_str((uncovered_items, sorted(list(accepted_keys))), nl=False)
>>> print(result)
([0, 9], ['a', 'c', 'd'])
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> candidate_sets_dict = {'a': [5, 3], 'b': [2, 3, 5],
...                        'c': [4, 8], 'd': [7, 6, 2, 1]}
>>> items = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> max_covers = 1
>>> tup = greedy_max_inden_setcover(candidate_sets_dict, items, max_covers)
>>> (uncovered_items, covered_items_list, accepted_keys) = tup
>>> result = ut.list_str((uncovered_items, sorted(list(accepted_keys))), nl=False)
>>> print(result)
([0, 3, 4, 5, 8, 9], ['d'])
utool.util_alg.group_indices(groupid_list)[source]

groups indicies of each item in groupid_list

Parameters:groupid_list (list) – list of group ids
SeeAlso:
vt.group_indices - optimized numpy version ut.apply_grouping

CommandLine:

python -m utool.util_alg --test-group_indices
python3 -m utool.util_alg --test-group_indices

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> groupid_list = ['b', 1, 'b', 1, 'b', 1, 'b', 'c', 'c', 'c', 'c']
>>> (keys, groupxs) = ut.group_indices(groupid_list)
>>> result = ut.repr3((keys, groupxs), nobraces=1, nl=1)
>>> print(result)
[1, 'b', 'c'],
[[1, 3, 5], [0, 2, 4, 6], [7, 8, 9, 10]],
utool.util_alg.inbounds(num, low, high, eq=False)[source]
Parameters:
  • num (scalar or ndarray) –
  • low (scalar or ndarray) –
  • high (scalar or ndarray) –
  • eq (bool) –
Returns:

is_inbounds
Return type:

scalar or ndarray

CommandLine:

python -m utool.util_alg --test-inbounds

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> num = np.array([[ 0.   ,  0.431,  0.279],
...                 [ 0.204,  0.352,  0.08 ],
...                 [ 0.107,  0.325,  0.179]])
>>> low  = .1
>>> high = .4
>>> eq = False
>>> is_inbounds = inbounds(num, low, high, eq)
>>> result = ut.numpy_str(is_inbounds)
>>> print(result)
np.array([[False, False,  True],
          [ True,  True, False],
          [ True,  True,  True]], dtype=bool)
utool.util_alg.is_prime(num)[source]

naive function for finding primes. Good for stress testing

References

http://thelivingpearl.com/2013/01/06/how-to-find-prime-numbers-in-python/

CommandLine:

python -m utool.util_alg --test-is_prime

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> with ut.Timer('isprime'):
>>>     series = [is_prime(n) for n in range(30)]
>>> result = ('primes = %s' % (str(ut.list_where(series[0:10])),))
>>> print(result)
primes = [2, 3, 5, 7]
utool.util_alg.item_hist(list_)[source]

counts the number of times each item appears in the dictionary

utool.util_alg.knapsack(items, maxweight, method=u'recursive')[source]

Solve the knapsack problem by finding the most valuable subsequence of items subject that weighs no more than maxweight.

Parameters:
  • items (tuple) – is a sequence of tuples (value, weight, id_), where value is a number and weight is a non-negative integer, and id_ is an item identifier.
  • maxweight (scalar) – is a non-negative integer.
Returns:

(total_value, items_subset) - a pair whose first element is the sum of values in the most valuable subsequence, and whose second element is the subsequence. Subset may be different depending on implementation (ie top-odwn recusrive vs bottom-up iterative)
Return type:

tuple

References

http://codereview.stackexchange.com/questions/20569/dynamic-programming-solution-to-knapsack-problem http://stackoverflow.com/questions/141779/solving-the-np-complete-problem-in-xkcd http://www.es.ele.tue.nl/education/5MC10/Solutions/knapsack.pdf

CommandLine:

python -m utool.util_alg --test-knapsack

python -m utool.util_alg --test-knapsack:0
python -m utool.util_alg --exec-knapsack:1

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> items = [(4, 12, 0), (2, 1, 1), (6, 4, 2), (1, 1, 3), (2, 2, 4)]
>>> maxweight = 15
>>> total_value, items_subset = knapsack(items, maxweight, method='recursive')
>>> total_value1, items_subset1 = knapsack(items, maxweight, method='iterative')
>>> result =  'total_value = %.2f\n' % (total_value,)
>>> result += 'items_subset = %r' % (items_subset,)
>>> print(result)
>>> ut.assert_eq(total_value1, total_value)
>>> ut.assert_eq(items_subset1, items_subset)
total_value = 11.00
items_subset = [(2, 1, 1), (6, 4, 2), (1, 1, 3), (2, 2, 4)]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> import utool as ut
>>> # Solve https://xkcd.com/287/
>>> weights = [2.15, 2.75, 3.35, 3.55, 4.2, 5.8] * 2
>>> items = [(w, w, i) for i, w in enumerate(weights)]
>>> maxweight = 15.05
>>> total_value, items_subset = knapsack(items, maxweight, method='recursive')
>>> total_value1, items_subset1 = knapsack(items, maxweight, method='iterative')
>>> total_weight = sum([t[1] for t in items_subset])
>>> print('total_weight = %r' % (total_weight,))
>>> result =  'total_value = %.2f' % (total_value,)
>>> print(result)
>>> print('items_subset = %r' % (items_subset,))
>>> print('items_subset1 = %r' % (items_subset1,))
>>> #assert items_subset1 == items_subset, 'NOT EQ\n%r !=\n%r' % (items_subset1, items_subset)
total_value = 15.05
Timeit:
>>> import utool as ut
>>> setup = ut.codeblock(
>>>     '''
        import utool as ut
        weights = [215, 275, 335, 355, 42, 58] * 10
        items = [(w, w, i) for i, w in enumerate(weights)]
        maxweight = 2505
        import numba
        knapsack_numba = numba.autojit(ut.knapsack_iterative)
        knapsack_numba = numba.autojit(ut.knapsack_iterative_numpy)
        ''')
>>> # Test load time
>>> stmt_list1 = ut.codeblock(
>>>     '''
        ut.knapsack_recursive(items, maxweight)
        ut.knapsack_iterative(items, maxweight)
        knapsack_numba(items, maxweight)
        ut.knapsack_iterative_numpy(items, maxweight)
        ''').split('\n')
>>> ut.util_dev.timeit_compare(stmt_list1, setup, int(5))
utool.util_alg.knapsack_greedy(items, maxweight)[source]

non-optimal greedy version of knapsack algorithm does not sort input. Sort the input by largest value first if desired.

Parameters:
  • items (tuple) – is a sequence of tuples (value, weight, id_), where value is a scalar and weight is a non-negative integer, and id_ is an item identifier.
  • maxweight (scalar) – is a non-negative integer.

CommandLine:

python -m utool.util_alg --exec-knapsack_greedy

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> items = [(4, 12, 0), (2, 1, 1), (6, 4, 2), (1, 1, 3), (2, 2, 4)]
>>> maxweight = 15
>>> total_value, items_subset = knapsack_greedy(items, maxweight)
>>> result =  'total_value = %r\n' % (total_value,)
>>> result += 'items_subset = %r' % (items_subset,)
>>> print(result)
total_value = 7
items_subset = [(4, 12, 0), (2, 1, 1), (1, 1, 3)]
utool.util_alg.knapsack_iterative(items, maxweight)[source]
utool.util_alg.knapsack_iterative_int(items, maxweight)[source]

Iterative knapsack method

Math:
maximize sum_{i in T} v_i subject to sum_{i in T} w_i leq W

Notes

dpmat is the dynamic programming memoization matrix. dpmat[i, w] is the total value of the items with weight at most W T is idx_subset, the set of indicies in the optimal solution

CommandLine:

python -m utool.util_alg --exec-knapsack_iterative_int --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> weights = [1, 3, 3, 5, 2, 1] * 2
>>> items = [(w, w, i) for i, w in enumerate(weights)]
>>> maxweight = 10
>>> total_value, items_subset = knapsack_iterative_int(items, maxweight)
>>> total_weight = sum([t[1] for t in items_subset])
>>> print('total_weight = %r' % (total_weight,))
>>> print('items_subset = %r' % (items_subset,))
>>> result =  'total_value = %.2f' % (total_value,)
>>> print(result)
total_value = 10.00
utool.util_alg.knapsack_iterative_numpy(items, maxweight)[source]

Iterative knapsack method

maximize sum_{i in T} v_i subject to sum_{i in T} w_i leq W

Notes

dpmat is the dynamic programming memoization matrix. dpmat[i, w] is the total value of the items with weight at most W T is the set of indicies in the optimal solution

utool.util_alg.knapsack_recursive(items, maxweight)[source]
utool.util_alg.max_size_max_distance_subset(items, min_thresh=0, Kstart=2, verbose=False)[source]
Parameters:
  • items –
  • min_thresh (int) – (default = 0)
  • Kstart (int) – (default = 2)
Returns:

prev_subset_idx
Return type:

?

CommandLine:

python -m utool.util_alg --exec-max_size_max_distance_subset

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> items = [1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> min_thresh = 3
>>> Kstart = 2
>>> verbose = True
>>> prev_subset_idx = max_size_max_distance_subset(items, min_thresh,
>>>                                                Kstart, verbose=verbose)
>>> result = ('prev_subset_idx = %s' % (str(prev_subset_idx),))
>>> print(result)
utool.util_alg.maximin_distance_subset1d(items, K=None, min_thresh=None, verbose=False)[source]

Greedy algorithm, may be exact for 1d case.

CommandLine:

python -m utool.util_alg --exec-maximin_distance_subset1d

Example

>>> # DISABLE_DOCTEST
>>> import utool as ut
>>> from utool.util_alg import *  # NOQA
>>> #items = [1, 2, 3, 4, 5, 6, 7]
>>> items = [20, 1, 1, 9, 21, 6, 22]
>>> min_thresh = 5
>>> K = None
>>> result = maximin_distance_subset1d(items, K, min_thresh, verbose=True)
>>> print(result)

Example

>>> # DISABLE_DOCTEST
>>> import utool as ut
>>> from utool.util_alg import *  # NOQA
>>> #items = [1, 2, 3, 4, 5, 6, 7]
>>> items = [0, 1]
>>> min_thresh = 5
>>> K = None
>>> result = maximin_distance_subset1d(items, K, min_thresh, verbose=True)
>>> print(result)
utool.util_alg.maximum_distance_subset(items, K, verbose=False)[source]

FIXME: I believe this does not work.

Returns a subset of size K from items with the maximum pairwise distance

References

stackoverflow.com/questions/12278528/subset-elements-furthest-apart-eachother stackoverflow.com/questions/13079563/condensed-distance-matrix-pdist

Recurance:
Let X[n,k] be the solution for selecting k elements from first n elements items. X[n, k] = max( max( X[m, k - 1] + (sum_{p in prev_solution} dist(o, p)) for o < n and o not in prev solution) ) for m < n.

Example

>>> import scipy.spatial.distance as spdist
>>> items = [1, 6, 20, 21, 22]

CommandLine:

python -m utool.util_alg --exec-maximum_distance_subset

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> #items = [1, 2, 3, 4, 5, 6, 7]
>>> items = [1, 6, 20, 21, 22]
>>> K = 3
>>> result = maximum_distance_subset(items, K)
>>> print(result)
(42.0, array([4, 3, 0]), array([22, 21,  1]))
utool.util_alg.negative_minclamp_inplace(arr)[source]
utool.util_alg.norm_zero_one(array, dim=None)[source]

normalizes a numpy array from 0 to 1 based in its extent

Parameters:
  • array (ndarray) –
  • dim (int) –
Returns:
Return type:

ndarray

CommandLine:

python -m utool.util_alg --test-norm_zero_one

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> array = np.array([ 22, 1, 3, 2, 10, 42, ])
>>> dim = None
>>> array_norm = norm_zero_one(array, dim)
>>> result = np.array_str(array_norm, precision=3)
>>> print(result)
[ 0.512  0.     0.049  0.024  0.22   1.   ]
utool.util_alg.normalize(array, dim=0)[source]
utool.util_alg.num_partitions(num_items)[source]
utool.util_alg.number_of_decimals(num)[source]
Parameters:num (float) –

References

stackoverflow.com/questions/6189956/finding-decimal-places

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> num = 15.05
>>> result = number_of_decimals(num)
>>> print(result)
2
utool.util_alg.product_nonsame(list1, list2)[source]

product of list1 and list2 where items are non equal

utool.util_alg.rad_to_deg(radians)[source]
utool.util_alg.safe_div(a, b)[source]
utool.util_alg.safe_pdist(arr, *args, **kwargs)[source]
Kwargs:
metric = ut.absdiff
SeeAlso:
scipy.spatial.distance.pdist

TODO: move to vtool

utool.util_alg.self_prodx(list_)[source]
utool.util_alg.solve_boolexpr()[source]

sudo pip install git+https://github.com/tpircher/quine-mccluskey.git sudo pip uninstall quine_mccluskey pip uninstall quine_mccluskey

utool.util_alg.square_pdist(arr, *args, **kwargs)[source]
utool.util_alg.triangular_number(n)[source]
Latex:
T_n = sum_{k=1}^{n} k = frac{n (n + 1)}{2} = binom{n + 1}{2}

References

en.wikipedia.org/wiki/Triangular_number

utool.util_alg.ungroup(grouped_items, groupxs, maxval)[source]

Ungroups items

SeeAlso:
vt.invert_apply_grouping
utool.util_alg.unixtime_hourdiff(x, y)[source]
Parameters:
  • x –
  • y (ndarray) – labels
Returns:
Return type:

?

CommandLine:

python -m utool.util_alg --exec-unixtime_hourdiff --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> x = np.nan
>>> y = 0
>>> result = unixtime_hourdiff(x, y)
>>> print(result)
>>> ut.quit_if_noshow()
>>> import plottool as pt
>>> ut.show_if_requested()
utool.util_alg.upper_diag_self_prodx(list_)[source]

upper diagnoal of cartesian product of self and self. Weird name. fixme

Parameters:list (list) –
Returns:
Return type:list

CommandLine:

python -m utool.util_alg --exec-upper_diag_self_prodx

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_alg import *  # NOQA
>>> list_ = [1, 2, 3]
>>> result = upper_diag_self_prodx(list_)
>>> print(result)
[(1, 2), (1, 3), (2, 3)]
utool.util_alg.xywh_to_tlbr(bbox, img_wh)[source]

converts xywh format to (tlx, tly, blx, bly)

utool.util_aliases module

Aliases harder to remember or very common standard module functions

utool.util_arg module

Handles command line parsing

class utool.util_arg.ArgumentParser2(parser)[source]

Bases: object

Wrapper around argparse.ArgumentParser with convinence functions

add_arg(switch, *args, **kwargs)[source]
add_argument_group(*args, **kwargs)[source]
add_flag(switch, default=False, **kwargs)[source]
add_float(switch, *args, **kwargs)[source]
add_int(switch, *args, **kwargs)[source]
add_intlist(switch, *args, **kwargs)[source]
add_ints(switch, *args, **kwargs)
add_meta(switch, type, default=None, help=u'', **kwargs)[source]
add_str(switch, *args, **kwargs)[source]
add_strlist(switch, *args, **kwargs)[source]
add_strs(switch, *args, **kwargs)
utool.util_arg.argparse_dict(default_dict_, lbl=None, verbose=None, only_specified=False, force_keys={}, type_hint=None, alias_dict={})[source]

Gets values for a dict based on the command line

Parameters:
  • default_dict –
  • only_specified (bool) – if True only returns keys that are specified on commandline. no defaults.
Returns:

dict_ - a dictionary
Return type:

dict_

CommandLine:

python -m utool.util_arg --test-argparse_dict
python -m utool.util_arg --test-argparse_dict --foo=3
python -m utool.util_arg --test-argparse_dict --flag1
python -m utool.util_arg --test-argparse_dict --flag2
python -m utool.util_arg --test-argparse_dict --noflag2
python -m utool.util_arg --test-argparse_dict --thresh=43
python -m utool.util_arg --test-argparse_dict --bins=-10
python -m utool.util_arg --test-argparse_dict --bins=-10 --only-specified --helpx

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> default_dict_ = {
...    'bins': 8,
...    'foo': None,
...    'flag1': False,
...    'flag2': True,
...    'max': 0.2,
...    'neg': -5,
...    'thresh': -5.333,
... }
>>> # execute function
>>> only_specified = ut.get_argflag('--only-specified')
>>> dict_ = argparse_dict(default_dict_, only_specified=only_specified)
>>> # verify results
>>> result = ut.dict_str(dict_, sorted_=True)
>>> print(result)
utool.util_arg.argv_flag_dec(*argin, **kwargs)[source]

Decorators which control program flow based on sys.argv the decorated function does not execute without its corresponding flag

Kwargs:
default, quiet, indent, default
ReturnKwargs:
alias_flags
utool.util_arg.argv_flag_dec_true(func, **kwargs)[source]
utool.util_arg.autogen_argparse2(dpath_list)[source]

FUNCTION IS NOT FULLY IMPLEMENTED CURRENTLY ONLY RETURNS LIST OF FLAGS THAT THE PROGRAM SILENTLY TAKES

Example

>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> dpath_list = [
...     ut.truepath('~/code/utool/utool'),
...     ut.truepath('~/code/ibeis/ibeis'),
...     ut.truepath('~/code/guitool/guitool'),
...     ut.truepath('~/code/vtool/vtool'),
...     ut.truepath('~/code/plottool/plottool'),
... ]
>>> flagtups_list = autogen_argparse2(dpath_list)
>>> flagtup_list_ = [ut.regex_replace('[)(\']','',tupstr) for tupstr in ut.flatten(flagtups_list)]
>>> flagtup_list = ut.flatten([tupstr.split(',') for tupstr in flagtup_list_])
>>> flagtup_set = set([tupstr.strip() for tupstr in flagtup_list if tupstr.find('=') == -1])
>>> print('\n'.join(flagtup_set))
utool.util_arg.autogen_argparse_block(extra_args=[])[source]

SHOULD TURN ANY REGISTERED ARGS INTO A A NEW PARSING CONFIG FILE FOR BETTER –help COMMANDS

import utool as ut __REGISTERED_ARGS__ = ut.util_arg.__REGISTERED_ARGS__

Parameters:extra_args (list) – (default = [])

CommandLine:

python -m utool.util_arg --test-autogen_argparse_block

Example

>>> # DISABLE_DOCTEST
>>> import utool as ut
>>> extra_args = []
>>> result = ut.autogen_argparse_block(extra_args)
>>> print(result)
utool.util_arg.get_arg(argstr_, type_=None, default=None, help_=None, smartcast=True, return_specified=None, argv=None, verbose=None, debug=None, return_was_specified=False)

Returns a value of an argument specified on the command line after some flag

Parameters:
  • argstr (str or tuple) – string or tuple of strings denoting the command line values to parse
  • type (None) – type of the variable to parse (default = None)
  • default (None) – (default = None)
  • help (None) – help for this argument (not fully integrated) (default = None)
  • smartcast (bool) – tries to be smart about casting the parsed strings (default = True)
  • return_specified (bool) – (default = False)
  • argv (None) – override sys.argv with custom command line vector (default = None)
TODO:
depricate return_was_specified

CommandLine:

python -m utool.util_arg --test-get_argval
python -m utool.util_arg --exec-get_argval:0
python -m utool.util_arg --exec-get_argval:1
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest="holy grail"
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest="42"
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest=42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest 42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest', float)]])" --quest 42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval(('--nAssign'), int)]])" --nAssign 42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval(('--test'), str)]])" --test
python -c "import utool; print([(type(x), x) for x in [utool.get_argval(('--test'), str)]])" --test "foobar is good" --youbar ok

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> import sys
>>> argv = ['--spam', 'eggs', '--quest=holy grail', '--ans=42', '--the-val=1,2,3']
>>> # specify a list of args and kwargs to get_argval
>>> argstr_kwargs_list = [
>>>     ('--spam',                    dict(type_=str, default=None, argv=argv)),
>>>     ('--quest',                   dict(type_=str, default=None, argv=argv)),
>>>     (('--ans', '--foo'),          dict(type_=int, default=None, argv=argv)),
>>>     (('--not-there', '--absent'), dict(argv=argv)),
>>>     ('--the_val',                 dict(type_=list, argv=argv)),
>>>     ('--the-val',                 dict(type_=list, argv=argv)),
>>> ]
>>> # Execute the command with for each of the test cases
>>> res_list = []
>>> argstr_list = ut.get_list_column(argstr_kwargs_list, 0)
>>> for argstr_, kwargs in argstr_kwargs_list:
>>>     res = get_argval(argstr_, **kwargs)
>>>     res_list.append(res)
>>> result = ut.dict_str(ut.odict(zip(argstr_list, res_list)))
>>> result = result.replace('u\'', '\'')  # hack
>>> print(result)
{
    '--spam': 'eggs',
    '--quest': 'holy grail',
    ('--ans', '--foo'): 42,
    ('--not-there', '--absent'): None,
    '--the_val': [1, 2, 3],
    '--the-val': [1, 2, 3],
}

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> import sys
>>> argv = ['--slice1', '::', '--slice2=4:', '--slice3=::4', '--slice4', '[1,2,3,4]', '--slice5=3']
>>> # specify a list of args and kwargs to get_argval
>>> argstr_kwargs_list = [
>>>     ('--slice1',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice2',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice3',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice4',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice5',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>> ]
>>> # Execute the command with for each of the test cases
>>> res_list = []
>>> argstr_list = ut.get_list_column(argstr_kwargs_list, 0)
>>> list1 = [1,3,5,7,9]
>>> import numpy as np
>>> list2 = np.array([[1, 2],[3,4],[5,6],[7,8],[9,1]])
>>> for argstr_, kwargs in argstr_kwargs_list:
>>>     res = get_argval(argstr_, **kwargs)
>>>     print('---')
>>>     print('res = %r' % (res,))
>>>     print('list1[%r=%r] = %r' % (argstr_, res, ut.take(list1, res),))
>>>     print('list2[%r=%r] = %r' % (argstr_, res, list2[res].tolist(),))
>>>     res_list.append(res)
>>> result = ut.dict_str(ut.odict(zip(argstr_list, res_list)))
>>> result = result.replace('u\'', '\'')  # hack
>>> print(result)
utool.util_arg.get_arg_dict(argv=None, prefix_list=[u'--'], type_hints={})[source]

Yet another way for parsing args

CommandLine:

python -m utool.util_arg --exec-get_arg_dict
python -m utool.util_arg --test-get_arg_dict

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> import shlex
>>> argv = shlex.split('--test-show_name --name=IBEIS_PZ_0303 --db testdb3 --save "~/latex/crall-candidacy-2015/figures/IBEIS_PZ_0303.jpg" --dpath figures --caption="Shadowed"  --figsize=11,3 --no-figtitle -t foo bar baz biz --notitle')
>>> arg_dict = ut.get_arg_dict(argv, prefix_list=['--', '-'], type_hints={'t': list})
>>> result = ut.dict_str(arg_dict)
>>> # verify results
>>> print(result)
{
    'caption': 'Shadowed',
    'db': 'testdb3',
    'dpath': 'figures',
    'figsize': '11,3',
    'name': 'IBEIS_PZ_0303',
    'no-figtitle': True,
    'notitle': True,
    'save': '~/latex/crall-candidacy-2015/figures/IBEIS_PZ_0303.jpg',
    't': ['foo', 'bar', 'baz', 'biz'],
    'test-show_name': True,
}
utool.util_arg.get_argflag(argstr_, default=False, help_=u'', return_specified=None, need_prefix=True, return_was_specified=False, argv=['/home/joncrall/venv/bin/sphinx-build', '-b', 'html', '-d', '_build/doctrees', '.', '_build/html'], **kwargs)[source]

Checks if the commandline has a flag or a corresponding noflag

Parameters:
  • argstr (str, list, or tuple) – the flag to look for
  • default (bool) – dont use this (default = False)
  • help (str) – a help string (default = ‘’)
  • return_specified (bool) – returns if flag was specified or not (default = False)
Returns:

(parsed_val, was_specified)
Return type:

tuple
TODO:
depricate return_was_specified

CommandLine:

python -m utool.util_arg --exec-get_argflag --noface --exec-mode
python -m utool.util_arg --exec-get_argflag --foo --exec-mode
python -m utool.util_arg --exec-get_argflag --no-foo --exec-mode
python -m utool.util_arg --exec-get_argflag --foo=True --exec-mode
python -m utool.util_arg --exec-get_argflag --foo=False  --exec-mode

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> argstr_ = '--foo'
>>> default = False
>>> help_ = ''
>>> return_specified = True
>>> (parsed_val, was_specified) = get_argflag(argstr_, default, help_, return_specified)
>>> result = ('(parsed_val, was_specified) = %s' % (str((parsed_val, was_specified)),))
>>> print(result)
utool.util_arg.get_argv_tail(scriptname, prefer_main=None, argv=['/home/joncrall/venv/bin/sphinx-build', '-b', 'html', '-d', '_build/doctrees', '.', '_build/html'])[source]

gets the rest of the arguments after a script has been invoked hack. accounts for python -m scripts.

Parameters:scriptname (str) –

CommandLine:

python -m utool.util_arg --test-get_argv_tail

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> from os.path import relpath, dirname
>>> scriptname = 'utool.util_arg'
>>> prefer_main = False
>>> argv=['python', '-m', 'utool.util_arg', '--test-get_argv_tail']
>>> tail = get_argv_tail(scriptname, prefer_main, argv)
>>> # hack
>>> tail[0] = ut.ensure_unixslash(relpath(tail[0], dirname(dirname(ut.__file__))))
>>> result = ut.repr2(tail)
>>> print(result)
['utool/util_arg.py', '--test-get_argv_tail']

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> from os.path import relpath, dirname
>>> scriptname = 'utprof.py'
>>> prefer_main = True
>>> argv=['utprof.py', '-m', 'utool', '--tf', 'get_argv_tail']
>>> tail = get_argv_tail(scriptname, prefer_main, argv)
>>> # hack
>>> tail[0] = ut.ensure_unixslash(relpath(tail[0], dirname(dirname(ut.__file__))))
>>> result = ut.repr2(tail)
>>> print(result)
['utool/__main__.py', '--tf', 'get_argv_tail']
utool.util_arg.get_argval(argstr_, type_=None, default=None, help_=None, smartcast=True, return_specified=None, argv=None, verbose=None, debug=None, return_was_specified=False)[source]

Returns a value of an argument specified on the command line after some flag

Parameters:
  • argstr (str or tuple) – string or tuple of strings denoting the command line values to parse
  • type (None) – type of the variable to parse (default = None)
  • default (None) – (default = None)
  • help (None) – help for this argument (not fully integrated) (default = None)
  • smartcast (bool) – tries to be smart about casting the parsed strings (default = True)
  • return_specified (bool) – (default = False)
  • argv (None) – override sys.argv with custom command line vector (default = None)
TODO:
depricate return_was_specified

CommandLine:

python -m utool.util_arg --test-get_argval
python -m utool.util_arg --exec-get_argval:0
python -m utool.util_arg --exec-get_argval:1
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest="holy grail"
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest="42"
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest=42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest')]])" --quest 42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval('--quest', float)]])" --quest 42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval(('--nAssign'), int)]])" --nAssign 42
python -c "import utool; print([(type(x), x) for x in [utool.get_argval(('--test'), str)]])" --test
python -c "import utool; print([(type(x), x) for x in [utool.get_argval(('--test'), str)]])" --test "foobar is good" --youbar ok

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> import sys
>>> argv = ['--spam', 'eggs', '--quest=holy grail', '--ans=42', '--the-val=1,2,3']
>>> # specify a list of args and kwargs to get_argval
>>> argstr_kwargs_list = [
>>>     ('--spam',                    dict(type_=str, default=None, argv=argv)),
>>>     ('--quest',                   dict(type_=str, default=None, argv=argv)),
>>>     (('--ans', '--foo'),          dict(type_=int, default=None, argv=argv)),
>>>     (('--not-there', '--absent'), dict(argv=argv)),
>>>     ('--the_val',                 dict(type_=list, argv=argv)),
>>>     ('--the-val',                 dict(type_=list, argv=argv)),
>>> ]
>>> # Execute the command with for each of the test cases
>>> res_list = []
>>> argstr_list = ut.get_list_column(argstr_kwargs_list, 0)
>>> for argstr_, kwargs in argstr_kwargs_list:
>>>     res = get_argval(argstr_, **kwargs)
>>>     res_list.append(res)
>>> result = ut.dict_str(ut.odict(zip(argstr_list, res_list)))
>>> result = result.replace('u\'', '\'')  # hack
>>> print(result)
{
    '--spam': 'eggs',
    '--quest': 'holy grail',
    ('--ans', '--foo'): 42,
    ('--not-there', '--absent'): None,
    '--the_val': [1, 2, 3],
    '--the-val': [1, 2, 3],
}

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> import sys
>>> argv = ['--slice1', '::', '--slice2=4:', '--slice3=::4', '--slice4', '[1,2,3,4]', '--slice5=3']
>>> # specify a list of args and kwargs to get_argval
>>> argstr_kwargs_list = [
>>>     ('--slice1',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice2',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice3',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice4',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>>     ('--slice5',            dict(type_='fuzzy_subset', default=None, argv=argv)),
>>> ]
>>> # Execute the command with for each of the test cases
>>> res_list = []
>>> argstr_list = ut.get_list_column(argstr_kwargs_list, 0)
>>> list1 = [1,3,5,7,9]
>>> import numpy as np
>>> list2 = np.array([[1, 2],[3,4],[5,6],[7,8],[9,1]])
>>> for argstr_, kwargs in argstr_kwargs_list:
>>>     res = get_argval(argstr_, **kwargs)
>>>     print('---')
>>>     print('res = %r' % (res,))
>>>     print('list1[%r=%r] = %r' % (argstr_, res, ut.take(list1, res),))
>>>     print('list2[%r=%r] = %r' % (argstr_, res, list2[res].tolist(),))
>>>     res_list.append(res)
>>> result = ut.dict_str(ut.odict(zip(argstr_list, res_list)))
>>> result = result.replace('u\'', '\'')  # hack
>>> print(result)
utool.util_arg.get_dict_vals_from_commandline(default_dict_, lbl=None, verbose=None, only_specified=False, force_keys={}, type_hint=None, alias_dict={})

Gets values for a dict based on the command line

Parameters:
  • default_dict –
  • only_specified (bool) – if True only returns keys that are specified on commandline. no defaults.
Returns:

dict_ - a dictionary
Return type:

dict_

CommandLine:

python -m utool.util_arg --test-argparse_dict
python -m utool.util_arg --test-argparse_dict --foo=3
python -m utool.util_arg --test-argparse_dict --flag1
python -m utool.util_arg --test-argparse_dict --flag2
python -m utool.util_arg --test-argparse_dict --noflag2
python -m utool.util_arg --test-argparse_dict --thresh=43
python -m utool.util_arg --test-argparse_dict --bins=-10
python -m utool.util_arg --test-argparse_dict --bins=-10 --only-specified --helpx

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> default_dict_ = {
...    'bins': 8,
...    'foo': None,
...    'flag1': False,
...    'flag2': True,
...    'max': 0.2,
...    'neg': -5,
...    'thresh': -5.333,
... }
>>> # execute function
>>> only_specified = ut.get_argflag('--only-specified')
>>> dict_ = argparse_dict(default_dict_, only_specified=only_specified)
>>> # verify results
>>> result = ut.dict_str(dict_, sorted_=True)
>>> print(result)
utool.util_arg.get_flag(argstr_, default=False, help_=u'', return_specified=None, need_prefix=True, return_was_specified=False, argv=['/home/joncrall/venv/bin/sphinx-build', '-b', 'html', '-d', '_build/doctrees', '.', '_build/html'], **kwargs)

Checks if the commandline has a flag or a corresponding noflag

Parameters:
  • argstr (str, list, or tuple) – the flag to look for
  • default (bool) – dont use this (default = False)
  • help (str) – a help string (default = ‘’)
  • return_specified (bool) – returns if flag was specified or not (default = False)
Returns:

(parsed_val, was_specified)
Return type:

tuple
TODO:
depricate return_was_specified

CommandLine:

python -m utool.util_arg --exec-get_argflag --noface --exec-mode
python -m utool.util_arg --exec-get_argflag --foo --exec-mode
python -m utool.util_arg --exec-get_argflag --no-foo --exec-mode
python -m utool.util_arg --exec-get_argflag --foo=True --exec-mode
python -m utool.util_arg --exec-get_argflag --foo=False  --exec-mode

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> argstr_ = '--foo'
>>> default = False
>>> help_ = ''
>>> return_specified = True
>>> (parsed_val, was_specified) = get_argflag(argstr_, default, help_, return_specified)
>>> result = ('(parsed_val, was_specified) = %s' % (str((parsed_val, was_specified)),))
>>> print(result)
utool.util_arg.get_fpath_args(arglist_=None, pat=u'*')[source]
utool.util_arg.get_module_verbosity_flags(*labels)[source]

checks for standard flags for enableing module specific verbosity

utool.util_arg.get_verbflag(*labels)

checks for standard flags for enableing module specific verbosity

utool.util_arg.make_argparse2(prog=u'Program', description=u'', *args, **kwargs)[source]
utool.util_arg.parse_arglist_hack(argx, argv=None)[source]
utool.util_arg.parse_cfgstr_list(cfgstr_list, smartcast=True, oldmode=True)[source]

Parses a list of items in the format [‘var1:val1’, ‘var2:val2’, ‘var3:val3’] the ‘=’ character can be used instead of the ‘:’ character if desired

Parameters:cfgstr_list (list) –
Returns:cfgdict
Return type:dict

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> cfgstr_list = ['var1=val1', 'var2=1', 'var3=1.0', 'var4=None', 'var5=[1,2,3]', 'var6=(a,b,c)']
>>> smartcast = True
>>> cfgdict = parse_cfgstr_list(cfgstr_list, smartcast, oldmode=False)
>>> result = ut.repr2(cfgdict, sorted_=True, newlines=False)
>>> print(result)
{'var1': 'val1', 'var2': 1, 'var3': 1.0, 'var4': None, 'var5': [1, 2, 3], 'var6': ('a', 'b', 'c')}

{‘var1’: ‘val1’, ‘var2’: 1, ‘var3’: 1.0, ‘var4’: None}

{‘var4’: None, ‘var1’: ‘val1’, ‘var3’: 1.0, ‘var2’: 1}

utool.util_arg.parse_dict_from_argv(default_dict_, lbl=None, verbose=None, only_specified=False, force_keys={}, type_hint=None, alias_dict={})

Gets values for a dict based on the command line

Parameters:
  • default_dict –
  • only_specified (bool) – if True only returns keys that are specified on commandline. no defaults.
Returns:

dict_ - a dictionary
Return type:

dict_

CommandLine:

python -m utool.util_arg --test-argparse_dict
python -m utool.util_arg --test-argparse_dict --foo=3
python -m utool.util_arg --test-argparse_dict --flag1
python -m utool.util_arg --test-argparse_dict --flag2
python -m utool.util_arg --test-argparse_dict --noflag2
python -m utool.util_arg --test-argparse_dict --thresh=43
python -m utool.util_arg --test-argparse_dict --bins=-10
python -m utool.util_arg --test-argparse_dict --bins=-10 --only-specified --helpx

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_arg import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> default_dict_ = {
...    'bins': 8,
...    'foo': None,
...    'flag1': False,
...    'flag2': True,
...    'max': 0.2,
...    'neg': -5,
...    'thresh': -5.333,
... }
>>> # execute function
>>> only_specified = ut.get_argflag('--only-specified')
>>> dict_ = argparse_dict(default_dict_, only_specified=only_specified)
>>> # verify results
>>> result = ut.dict_str(dict_, sorted_=True)
>>> print(result)
utool.util_arg.reset_argrecord()[source]

forgets about the args already parsed

utool.util_arg.switch_sanataize(switch)[source]

utool.util_assert module

utool.util_assert.assert_all_eq(item_list, eq_=<built-in function eq>)[source]
utool.util_assert.assert_all_in(key_list, valid_list, msg='')[source]
utool.util_assert.assert_all_not_None(list_, list_name='some_list', key_list=[], verbose=True, veryverbose=False)[source]
utool.util_assert.assert_almost_eq(arr_test, arr_target, thresh=1e-11)[source]
Parameters:
  • arr_test (ndarray or list) –
  • arr_target (ndarray or list) –
  • thresh (scalar or ndarray or list) –
utool.util_assert.assert_eq(var1, var2, msg='', var1_name=None, var2_name=None, verbose=True)[source]
utool.util_assert.assert_eq_len(list1, list2, additional_msg='')
utool.util_assert.assert_inbounds(num, low, high, msg='', eq=False, verbose=True)[source]
Parameters:
  • num (scalar) –
  • low (scalar) –
  • high (scalar) –
  • msg (str) –
utool.util_assert.assert_lessthan(arr_test, arr_max, msg='')[source]
Parameters:
  • arr_test (ndarray or list) –
  • arr_target (ndarray or list) –
  • thresh (scalar or ndarray or list) –
utool.util_assert.assert_lists_eq(list1, list2, failmsg='', verbose=False)[source]
utool.util_assert.assert_same_len(list1, list2, additional_msg='')[source]
utool.util_assert.assert_scalar_list(list_)[source]
utool.util_assert.assert_unflat_level(unflat_list, level=1, basetype=None)[source]
utool.util_assert.get_first_None_position(list_)[source]
utool.util_assert.lists_eq(list1, list2)[source]

recursive

utool.util_autogen module

class utool.util_autogen.PythonStatement(stmt)[source]

Bases: object

Thin wrapper around a string representing executable python code

utool.util_autogen.auto_docstr(modname, funcname, verbose=True, moddir=None, **kwargs)[source]

called from vim. Uses strings of filename and modnames to build docstr

Parameters:
  • modname (str) – name of a python module
  • funcname (str) – name of a function in the module
Returns:

docstr
Return type:

str

CommandLine:

python -m utool.util_autogen --exec-auto_docstr
python -m utool --tf auto_docstr

Example

>>> import utool as ut
>>> from utool.util_autogen import *  # NOQA
>>> ut.util_autogen.rrr(verbose=False)
>>> #docstr = ut.auto_docstr('ibeis.algo.hots.smk.smk_index', 'compute_negentropy_names')
>>> modname = ut.get_argval('--modname', default='utool.util_autogen')
>>> funcname = ut.get_argval('--funcname', default='auto_docstr')
>>> moddir = ut.get_argval('--moddir', type_=str, default=None)
>>> docstr = ut.util_autogen.auto_docstr(modname, funcname)
>>> print(docstr)
utool.util_autogen.autofix_codeblock(codeblock, max_line_len=80, aggressive=False, very_aggressive=False, experimental=False)[source]

Uses autopep8 to format a block of code

Example

>>> import utool as ut
>>> codeblock = ut.codeblock(
    '''
    def func( with , some = 'Problems' ):

syntax =’Ok’ but = ‘Its very messy’ if None:

# syntax might not be perfect due to being cut off ommiting_this_line_still_works= True

‘’‘)

>>> fixed_codeblock = ut.autofix_codeblock(codeblock)
>>> print(fixed_codeblock)
utool.util_autogen.dump_autogen_code(fpath, autogen_text, codetype='python', fullprint=None)[source]

Helper that write a file if -w is given on command line, otherwise it just prints it out. It has the opption of comparing a diff to the file.

utool.util_autogen.find_modname_in_pythonpath(modname)[source]
utool.util_autogen.is_modname_in_pythonpath(modname)[source]
utool.util_autogen.load_func_from_module(modname, funcname, verbose=True, moddir=None)[source]
Parameters:
  • modname (str) – module name
  • funcname (str) – function name
  • verbose (bool) – verbosity flag(default = True)
  • moddir (None) – (default = None)

CommandLine:

python -m utool.util_autogen --exec-load_func_from_module

Example

>>> # UNSTABLE_DOCTEST
>>> from utool.util_autogen import *  # NOQA
>>> import utool as ut
>>> #modname = 'plottool.plots'
>>> #funcname = 'multi_plot'
>>> modname = 'utool.util_path'
>>> funcname = 'checkpath'
>>> verbose = True
>>> moddir = None
>>> func, module, error_str = load_func_from_module(modname, funcname, verbose, moddir)
>>> source = ut.get_func_sourcecode(func, strip_docstr=True, strip_comments=True)
>>> keyname = ut.named_field('keyname', ut.REGEX_VARNAME)
>>> default = ut.named_field('default', '[\'\"A-Za-z_][A-Za-z0-9_\'\"]*')
>>> pattern = re.escape('kwargs.get(\'') + keyname + re.escape('\',')
>>> kwarg_keys = [match.groupdict()['keyname'] for match in re.finditer(pattern, source)]
utool.util_autogen.make_args_docstr(argname_list, argtype_list, argdesc_list, ismethod)[source]

Builds the argument docstring

Parameters:
  • argname_list (list) – names
  • argtype_list (list) – types
  • argdesc_list (list) – descriptions
Returns:

arg_docstr
Return type:

str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_autogen import *  # NOQA
>>> argname_list = ['argname_list', 'argtype_list', 'argdesc_list']
>>> argtype_list = ['list', 'list', 'list']
>>> argdesc_list = ['names', 'types', 'descriptions']
>>> ismethod = False
>>> arg_docstr = make_args_docstr(argname_list, argtype_list, argdesc_list, ismethod)
>>> result = str(arg_docstr)
>>> print(result)
argname_list (list): names
argtype_list (list): types
argdesc_list (list): descriptions
utool.util_autogen.make_cmdline_docstr(funcname, modname)[source]
utool.util_autogen.make_default_docstr(func, with_args=True, with_ret=True, with_commandline=True, with_example=True, with_header=False, with_debug=False)[source]

Tries to make a sensible default docstr so the user can fill things in without typing too much

# TODO: Interleave old documentation with new documentation

Parameters:
  • func (function) – live python function
  • with_args (bool) –
  • with_ret (bool) – (default = True)
  • with_commandline (bool) – (default = True)
  • with_example (bool) – (default = True)
  • with_header (bool) – (default = False)
  • with_debug (bool) – (default = False)
Returns:

(argname, val)
Return type:

tuple

CommandLine:

python -m utool.util_autogen --exec-make_default_docstr --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_autogen import *  # NOQA
>>> import utool as ut
>>> func = ut.make_default_docstr
>>> #func = ut.make_args_docstr
>>> func = PythonStatement
>>> default_docstr = make_default_docstr(func)
>>> result = str(default_docstr)
>>> print(result)
utool.util_autogen.make_default_module_maintest(modname, modpath=None)[source]

TODO: use path relative to home dir if the file is a script

Parameters:modname (str) – module name
Returns:text source code
Return type:str

CommandLine:

python -m utool.util_autogen --test-make_default_module_maintest

References

http://legacy.python.org/dev/peps/pep-0338/

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_autogen import *  # NOQA
>>> modname = 'utool.util_autogen'
>>> text = make_default_module_maintest(modname)
>>> result = str(text)
>>> print(result)
utool.util_autogen.make_docstr_block(header, block)[source]
utool.util_autogen.make_example_docstr(funcname=None, modname=None, argname_list=None, defaults=None, return_type=None, return_name=None, ismethod=False)[source]

Creates skeleton code to build an example doctest

Parameters:
  • funcname (str) – function name
  • modname (str) – module name
  • argname_list (str) – list of argument names
  • defaults (None) –
  • return_type (None) –
  • return_name (str) – return variable name
  • ismethod (bool) –
Returns:

examplecode
Return type:

str

CommandLine:

python -m utool.util_autogen --test-make_example_docstr

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_autogen import *  # NOQA
>>> # build test data
>>> funcname = 'make_example_docstr'
>>> modname = 'utool.util_autogen'
>>> argname_list = ['qaids', 'qreq_']
>>> defaults = None
>>> return_type = tuple
>>> return_name = 'foo'
>>> ismethod = False
>>> # execute function
>>> examplecode = make_example_docstr(funcname, modname, argname_list, defaults, return_type, return_name, ismethod)
>>> # verify results
>>> result = str(examplecode)
>>> print(result)
# DISABLE_DOCTEST
from utool.util_autogen import *  # NOQA
import utool as ut
import ibeis
species = ibeis.const.TEST_SPECIES.ZEB_PLAIN
qaids = ibs.get_valid_aids(species=species)
qreq_ = ibeis.testdata_qreq_()
foo = make_example_docstr(qaids, qreq_)
result = ('foo = %s' % (ut.repr2(foo),))
print(result)
utool.util_autogen.make_returns_or_yeilds_docstr(return_type, return_name, return_desc)[source]
utool.util_autogen.makeinit(mod_dpath, exclude_modnames=[], use_star=False)[source]
Parameters:
  • mod_dpath (str) –
  • exclude_modnames (list) – (default = [])
  • use_star (bool) – (default = False)
Returns:

init_codeblock
Return type:

str

CommandLine:

python -m utool.util_autogen --exec-makeinit --modname=ibeis.algo

Example

>>> # SCRIPT
>>> from utool.util_autogen import *  # NOQA
>>> import utool as ut
>>> modname = ut.get_argval('--modname', str, default=None)
>>> mod_dpath = (os.getcwd() if modname is None else
>>>              ut.get_modpath_from_modname(modname, prefer_pkg=True))
>>> mod_dpath = ut.unixpath(mod_dpath)
>>> mod_fpath = join(mod_dpath, '__init__.py')
>>> exclude_modnames = ut.get_argval(('--exclude', '-x'), list, default=[])
>>> use_star = ut.get_argflag('--star')
>>> init_codeblock = makeinit(mod_dpath, exclude_modnames, use_star)
>>> ut.dump_autogen_code(mod_fpath, init_codeblock)
utool.util_autogen.print_auto_docstr(modname, funcname)[source]

python -c “import utool; utool.print_auto_docstr(‘ibeis.algo.hots.smk.smk_index’, ‘compute_negentropy_names’)” python -c “import utool; utool.print_auto_docstr(‘ibeis.algo.hots.smk.smk_index’, ‘compute_negentropy_names’)”

utool.util_autogen.remove_codeblock_syntax_sentinals(code_text)[source]

Removes template comments and vim sentinals

Parameters:code_text (str) –
Returns:code_text_
Return type:str
utool.util_autogen.write_modscript_alias(fpath, modname, args='', pyscript='python')[source]

convinience function because $@ is annoying to paste into the terminal

utool.util_cache module

class utool.util_cache.Cachable[source]

Bases: object

Abstract base class.

This class which enables easy caching of object dictionarys

must implement get_cfgstr()

delete(cachedir=None, cfgstr=None, verbose=True)[source]

saves query result to directory

ext = u'.cPkl'
fuzzyload(cachedir=None, partial_cfgstr=u'', **kwargs)[source]

Try and load from a partially specified configuration string

get_cachedir(cachedir=None)[source]
get_cfgstr()[source]
get_fname(cfgstr=None, ext=None)[source]
get_fpath(cachedir=None, cfgstr=None, ext=None)[source]
get_prefix()[source]
glob_valid_targets(cachedir=None, partial_cfgstr=u'')[source]
load(cachedir=None, cfgstr=None, fpath=None, verbose=None, quiet=False, ignore_keys=None)[source]

Loads the result from the given database

save(cachedir=None, cfgstr=None, verbose=False, quiet=False, ignore_keys=None)[source]

saves query result to directory

exception utool.util_cache.CacheMissException[source]

Bases: exceptions.Exception

class utool.util_cache.Cacher(fname, cfgstr=None, cache_dir=u'default', appname=u'utool', verbose=None)[source]

Bases: object

old non inhertable version of cachable

load(cfgstr=None)[source]
save(data, cfgstr=None)[source]
tryload(cfgstr=None)[source]
class utool.util_cache.GlobalShelfContext(appname)[source]

Bases: object

older class. might need update

class utool.util_cache.KeyedDefaultDict(default_func, *args, **kwargs)[source]

Bases: utool.util_dict.DictLike

getitem(key)[source]
keys()[source]
setitem(key, value)[source]
values()[source]
class utool.util_cache.LRUDict(max_size)[source]

Bases: object

Pure python implementation for lru cache fallback

References

http://www.kunxi.org/blog/2014/05/lru-cache-in-python/

Parameters:max_size (int) – (default = 5)
Returns:cache_obj
Return type:LRUDict

CommandLine:

python -m utool.util_cache --test-LRUDict

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_cache import *  # NOQA
>>> max_size = 5
>>> self = LRUDict(max_size)
>>> for count in range(0, 5):
...     self[count] = count
>>> print(self)
>>> self[0]
>>> for count in range(5, 8):
...     self[count] = count
>>> print(self)
>>> del self[5]
>>> assert 4 in self
>>> result = ('self = %r' % (self,))
>>> print(result)
self = LRUDict({
    4: 4,
    0: 0,
    6: 6,
    7: 7,
})
clear()[source]
has_key(item)[source]
items()[source]
iteritems()[source]
iterkeys()[source]
itervalues()[source]
keys()[source]
values()[source]
class utool.util_cache.LazyDict(other=None, is_eager=True, verbose=False, **kwargs)[source]

Bases: object

Hacky dictionary where values that are functions are counted as lazy

CommandLine:

python -m utool.util_cache --exec-LazyDict

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_cache import *  # NOQA
>>> import utool as ut
>>> self = ut.LazyDict()
>>> self['foo'] = lambda: 5
>>> self['bar'] = 4
>>> try:
>>>     self['foo'] = lambda: 9
>>>     assert False, 'should not be able to override computable functions'
>>> except ValueError:
>>>     pass
>>> self['biz'] = lambda: 9
>>> d = {}
>>> d.update(**self)
>>> self['spam'] = lambda: 'eggs'
>>> self.printinfo()
>>> print(self.tostring(is_eager=False))
all_keys()[source]
asdict(is_eager=None)[source]
cached_keys()[source]

only keys whose vals that have been explicitly set without a backup func

clear_evaluated()[source]
clear_stored(keys=None)[source]
eager_eval(key)[source]
evaluated_keys()[source]

only keys whose vals have been evaluated from a stored function

get(key, *d)[source]
getitem(key, is_eager=None)[source]
items()[source]
keys()[source]
lazy_eval(key)[source]
nocache_eval(key)[source]

forces function evaluation

nonreconstructable_keys()[source]

only keys whose vals that have been explicitly set without a backup func

printinfo()[source]
reconstructable_keys()[source]

only keys whose vals that have been set with a backup func

rrr(verbose=True)

special class reloading function

set_lazy_func(key, func)[source]
setitem(key, value)[source]
stored_keys()[source]

keys whose vals that have been explicitly set or evaluated

tostring(is_eager=None, keys=None, **kwargs)[source]
unevaluated_keys()[source]

keys whose vals can be constructed but have not been

update(dict_, **kwargs)[source]
values()[source]
class utool.util_cache.LazyList(**kwargs)[source]

Bases: object

very hacky list implemented as a dictionary

append(item)[source]
rrr(verbose=True)

special class reloading function

tolist()[source]
class utool.util_cache.ShelfCacher(fpath, enabled=True)[source]

Bases: object

yet another cacher

clear()[source]
close()[source]
keys()[source]
load(cachekey)[source]
rrr(verbose=True)

special class reloading function

save(cachekey, data)[source]
utool.util_cache.cached_func(fname=None, cache_dir=u'default', appname=u'utool', key_argx=None, key_kwds=None, use_cache=None, verbose=None)[source]

Wraps a function with a Cacher object

uses a hash of arguments as input

Parameters:
  • fname (str) – file name (defaults to function name)
  • cache_dir (unicode) – (default = u’default’)
  • appname (unicode) – (default = u’utool’)
  • key_argx (None) – (default = None)
  • key_kwds (None) – (default = None)
  • use_cache (bool) – turns on disk based caching(default = None)

CommandLine:

python -m utool.util_cache --exec-cached_func

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> def costly_func(a, b, c='d', *args, **kwargs):
...    return ([a] * b, c, args, kwargs)
>>> ans0 = costly_func(41, 3)
>>> ans1 = costly_func(42, 3)
>>> closure_ = ut.cached_func('costly_func', appname='utool_test',
>>>         key_argx=[0, 1])
>>> efficient_func = closure_(costly_func)
>>> ans2 = efficient_func(42, 3)
>>> ans3 = efficient_func(42, 3)
>>> ans4 = efficient_func(41, 3)
>>> ans5 = efficient_func(41, 3)
>>> assert ans1 == ans2
>>> assert ans2 == ans3
>>> assert ans5 == ans4
>>> assert ans5 == ans0
>>> assert ans1 != ans0
utool.util_cache.cachestr_repr(val)[source]

Representation of an object as a cache string.

utool.util_cache.consensed_cfgstr(prefix, cfgstr, max_len=128, cfgstr_hashlen=16)[source]
utool.util_cache.delete_global_cache(appname=u'default')[source]

Reads cache files to a safe place in each operating system

utool.util_cache.from_json(json_str, allow_pickle=False)[source]

Decodes a JSON object specified in the utool convention

utool.util_cache.get_cfgstr_from_args(func, args, kwargs, key_argx, key_kwds, kwdefaults, argnames, use_hash=None)[source]
Dev:
argx = [‘fdsf’, ‘432443432432’, 43423432, ‘fdsfsd’, 3.2, True] memlist = list(map(cachestr_repr, argx))

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_cache import *  # NOQA
>>> use_hash = None
utool.util_cache.get_default_appname()[source]
utool.util_cache.get_func_result_cachekey(func_, args_=(), kwargs_={})[source]

TODO: recursive partial definitions kwargs = {} args = ([],)

utool.util_cache.get_global_cache_dir(appname=u'default', ensure=False)[source]

Returns (usually) writable directory for an application cache

utool.util_cache.get_global_shelf_fpath(appname=u'default', ensure=False)[source]

Returns the filepath to the global shelf

utool.util_cache.get_lru_cache(max_size=5)[source]
Parameters:max_size (int) –

References

https://github.com/amitdev/lru-dict

CommandLine:

python -m utool.util_cache --test-get_lru_cache

Example

>>> # UNSTABLE_DOCTEST
>>> from utool.util_cache import *  # NOQA
>>> import utool as ut  # NOQA
>>> max_size = 5
>>> # execute function
>>> cache_obj = get_lru_cache(max_size)
>>> cache_obj[1] = 1
>>> cache_obj[2] = 2
>>> cache_obj[3] = 3
>>> cache_obj[4] = 4
>>> cache_obj[5] = 5
>>> cache_obj[6] = 6
>>> # verify results
>>> result = ut.repr2(dict(cache_obj), nl=False)
>>> print(result)
{2: 2, 3: 3, 4: 4, 5: 5, 6: 6}
utool.util_cache.global_cache_dump(appname=u'default')[source]
utool.util_cache.global_cache_read(key, appname=u'default', **kwargs)[source]
utool.util_cache.global_cache_write(key, val, appname=u'default')[source]

Writes cache files to a safe place in each operating system

utool.util_cache.load_cache(dpath, fname, cfgstr, verbose=None)[source]
Parameters:
  • dpath (str) –
  • fname (str) –
  • cfgstr (str) –
Returns:

data
Return type:

pickleable

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_cache import *  # NOQA
>>> dpath = '?'
>>> fname = '?'
>>> cfgstr = '?'
>>> data = load_cache(dpath, fname, cfgstr)
>>> result = str(data)
>>> print(result)
utool.util_cache.make_utool_json_encoder(allow_pickle=False)[source]

References

http://stackoverflow.com/questions/8230315/python-sets-are http://stackoverflow.com/questions/11561932/why-does-json https://github.com/jsonpickle/jsonpickle http://stackoverflow.com/questions/24369666/typeerror-b1 http://stackoverflow.com/questions/30469575/how-to-pickle

utool.util_cache.save_cache(dpath, fname, cfgstr, data, verbose=None)[source]
Parameters:
  • dpath (str) –
  • fname (str) –
  • cfgstr (str) –
  • data (pickleable) –
Returns:

fpath
Return type:

str

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_cache import *  # NOQA
>>> import utool as ut
>>> dpath = ut.get_app_resource_dir('utool')
>>> fname = 'foo'
>>> cfgstr = 'bar'
>>> data = object()
>>> fpath = save_cache(dpath, fname, cfgstr, data)
>>> result = str(fpath)
>>> print(result)
utool.util_cache.shelf_open(fpath)[source]

allows for shelf to be used in with statements

References

http://stackoverflow.com/questions/7489732/easiest-way-to-add-a-function-to-existing-class

CommandLine:

python -m utool.util_cache --test-shelf_open

Example

>>> # UNSTABLE_DOCTEST
>>> import utool as ut
>>> fpath = ut.unixjoin(ut.ensure_app_resource_dir('utool'), 'testshelf.shelf')
>>> with ut.shelf_open(fpath) as dict_:
...     print(ut.dict_str(dict_))
utool.util_cache.text_dict_read(fpath)[source]
utool.util_cache.text_dict_write(fpath, dict_)[source]

Very naive, but readable way of storing a dictionary on disk FIXME: This broke on RoseMary’s big dataset. Not sure why. It gave bad syntax. And the SyntaxError did not seem to be excepted.

utool.util_cache.time_different_diskstores()[source]

%timeit shelf_write_test() # 15.1 ms per loop %timeit cPickle_write_test() # 1.26 ms per loop

%timeit shelf_read_test() # 8.77 ms per loop %timeit cPickle_read_test() # 2.4 ms per loop %timeit cPickle_read_test2() # 2.35 ms per loop

%timeit json_read_test() %timeit json_write_test()

utool.util_cache.to_json(val, allow_pickle=False)[source]

Converts a python object to a JSON string using the utool convention

Parameters:val (object) –
Returns:json_str
Return type:str

References

http://stackoverflow.com/questions/11561932/why-does-json-dumpslistnp

CommandLine:

python -m utool.util_cache --test-to_json
python3 -m utool.util_cache --test-to_json

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_cache import *  # NOQA
>>> import utool as ut
>>> import numpy as np
>>> import uuid
>>> val = [
>>>    '{"foo": "not a dict"}',
>>>    1.3,
>>>    [1],
>>>    slice(1, None, 1),
>>>    b'an ascii string',
>>>    np.array([1, 2, 3]),
>>>    ut.get_zero_uuid(),
>>>    ut.LazyDict(x='fo'),
>>>    ut.LazyDict,
>>> ]
>>> #val = ut.LazyDict(x='fo')
>>> json_str = ut.to_json(val, allow_pickle=True)
>>> result = ut.repr3(json_str)
>>> reload_val = ut.from_json(json_str, allow_pickle=True)
>>> # Make sure pickle doesnt happen by default
>>> try:
>>>     json_str = ut.to_json(val)
>>>     assert False, 'expected a type error'
>>> except TypeError:
>>>     print('Correctly got type error')
>>> try:
>>>     json_str = ut.from_json(val)
>>>     assert False, 'expected a type error'
>>> except TypeError:
>>>     print('Correctly got type error')
>>> print(result)
>>> print('original = ' + ut.repr3(val, nl=1))
>>> print('reconstructed = ' + ut.repr3(reload_val, nl=1))
utool.util_cache.tryload_cache(dpath, fname, cfgstr, verbose=None)[source]

returns None if cache cannot be loaded

utool.util_cache.tryload_cache_list(dpath, fname, cfgstr_list, verbose=False)[source]

loads a list of similar cached datas. Returns flags that needs to be computed

utool.util_cache.tryload_cache_list_with_compute(use_cache, dpath, fname, cfgstr_list, compute_fn, *args)[source]

tries to load data, but computes it if it can’t give a compute function

utool.util_cache.view_global_cache_dir(appname=u'default')[source]

utool.util_class module

In this module:
  • a metaclass allowing for reloading of single class instances
  • functions to autoinject methods into a class upon instance creation.
  • A wrapper class allowing an object’s properties to be used as kwargs
  • a metaclass to forward properties to another class

ReloadingMetaclass KwargsWrapper

class utool.util_class.HashComparable[source]

Bases: object

class utool.util_class.HashComparableMetaclass[source]

Bases: type

Defines extra methods for Configs

class utool.util_class.KwargsWrapper(obj)[source]

Bases: _abcoll.Mapping

Allows an arbitrary object attributes to be passed as a **kwargs argument

class utool.util_class.ReloadingMetaclass(metaself, name, bases, dct)[source]

Bases: type

Classes with this metaclass will be able to reload themselves on a per-instance basis using the rrr function.

If the functions _on_reload and _initialize_self exist they will be called after and before reload respectively. Any inject_instance functions should be handled there.

SeeAlso:
test_reloading_metaclass - shows a working example of this doctest

Example

>>> # DIABLE_DOCTEST
>>> from utool.util_class import *  # NOQA
>>> import utool as ut
>>> @six.add_metaclass(ut.ReloadingMetaclass)
>>> class Foo(object):
...     def __init__(self):
...         pass
>>> # You can edit foo on disk and call rrr in ipython
>>> # if you add a new function to it
>>> foo = Foo()
>>> # This will not work as a doctests because
>>> # Foo's parent module will be __main__ but
>>> # there will be no easy way to write to it.
>>> # This does work when you run from ipython
>>> @six.add_metaclass(ut.ReloadingMetaclass)
>>> class Foo(object):
...     def __init__(self):
...         pass
...     def bar(self):
...         return 'spam'
>>> foo.rrr()
>>> result = foo.bar()
>>> print(result)
spam
utool.util_class.autogen_explicit_injectable_metaclass(classname, regen_command=None, conditional_imports=None)[source]
Parameters:classname –
Returns:
Return type:?

CommandLine:

python -m utool.util_class --exec-autogen_explicit_injectable_metaclass

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_class import *  # NOQA
>>> from utool.util_class import  __CLASSTYPE_ATTRIBUTES__  # NOQA
>>> import ibeis
>>> import ibeis.control.IBEISControl
>>> classname = ibeis.control.controller_inject.CONTROLLER_CLASSNAME
>>> result = autogen_explicit_injectable_metaclass(classname)
>>> print(result)
utool.util_class.autogen_import_list(classname, conditional_imports=None)[source]
utool.util_class.decorate_class_method(func, classkey=None, skipmain=False)[source]

Will inject all decorated function as methods of classkey

classkey is some identifying string, tuple, or object

func can also be a tuple

utool.util_class.decorate_postinject(func, classkey=None, skipmain=False)[source]

Will perform func with argument self after inject_instance is called on classkey

classkey is some identifying string, tuple, or object

utool.util_class.get_classname(class_, local=False)[source]
Parameters:
  • class (type) –
  • local (bool) – (default = False)
Returns:

classname
Return type:

str

CommandLine:

python -m utool.util_class --exec-get_classname --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_class import *  # NOQA
>>> import utool as ut
>>> class_ = ReloadingMetaclass
>>> local = False
>>> assert get_classname(class_, local) == 'utool.util_class.ReloadingMetaclass'
>>> assert get_classname(class_, local=True) == 'ReloadingMetaclass'
utool.util_class.get_comparison_methods()[source]

makes methods for >, <, =, etc...

utool.util_class.get_injected_modules(classname)[source]

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_class import __CLASSNAME_CLASSKEY_REGISTER__  # NOQA
utool.util_class.inject_all_external_modules(self, classname=None, allow_override='override+warn', strict=True)[source]

dynamically injects registered module methods into a class instance

FIXME: naming convention and use this in all places where this clas is used

utool.util_class.inject_func_as_method(self, func, method_name=None, class_=None, allow_override=False, allow_main=False, verbose=True)[source]

Injects a function into an object as a method

Wraps func as a bound method of self. Then injects func into self It is preferable to use make_class_method_decorator and inject_instance

Parameters:
  • self (object) – class instance
  • func – some function whos first arugment is a class instance
  • method_name (str) – default=func.__name__, if specified renames the method
  • class (type) – if func is an unbound method of this class

References

http://stackoverflow.com/questions/1015307/python-bind-an-unbound-method

utool.util_class.inject_instance(self, classkey=None, allow_override=False, verbose=False, strict=True)[source]

Injects an instance (self) of type (classkey) with all functions registered to (classkey)

call this in the __init__ class function

Parameters:
  • self – the class instance
  • classkey – key for a class, preferably the class type itself, but it doesnt have to be
SeeAlso:
make_class_method_decorator

Example

>>> # DOCTEST_DISABLE
>>> utool.make_class_method_decorator(InvertedIndex)(smk_debug.invindex_dbgstr)
>>> utool.inject_instance(invindex)
utool.util_class.makeForwardingMetaclass(forwarding_dest_getter, whitelist, base_class=<type 'object'>)[source]

makes a metaclass that overrides __getattr__ and __setattr__ to forward some specific attribute references to a specified instance variable

utool.util_class.make_class_method_decorator(classkey, modname=None)[source]

register a class to be injectable classkey is a key that identifies the injected class REMEMBER to call inject_instance in __init__

Parameters:
  • classkey – the class to be injected into
  • modname – the global __name__ of the module youa re injecting from
Returns:

closure_decorate_class_method – decorator for injectable methods
Return type:

func

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> class CheeseShop(object):
...    def __init__(self):
...        import utool as ut
...        ut.inject_all_external_modules(self)
>>> cheeseshop_method = ut.make_class_method_decorator(CheeseShop)
>>> shop1 = CheeseShop()
>>> assert not hasattr(shop1, 'has_cheese'), 'have not injected yet'
>>> @cheeseshop_method
>>> def has_cheese(self):
>>>     return False
>>> shop2 = CheeseShop()
>>> assert shop2.has_cheese() is False, 'external method not injected'
>>> print('Cheese shop does not have cheese. All is well.')
utool.util_class.make_class_postinject_decorator(classkey, modname=None)[source]
Parameters:
  • classkey – the class to be injected into
  • modname – the global __name__ of the module youa re injecting from
Returns:

closure_decorate_postinject – decorator for injectable methods
Return type:

func
SeeAlso:
make_class_method_decorator
utool.util_class.postinject_instance(self, classkey, verbose=False)[source]
utool.util_class.private_rrr_factory()[source]
utool.util_class.reload_class_methods(self, class_, verbose=True)[source]

rebinds all class methods

Parameters:
  • self (object) – class instance to reload
  • class (type) – type to reload as

Example

>>> from utool.util_class import *  # NOQA
>>> self = '?'
>>> class_ = '?'
>>> result = reload_class_methods(self, class_)
>>> print(result)
utool.util_class.reload_injected_modules(classname)[source]
utool.util_class.reloadable_class(cls)[source]

convinience decorator instead of @six.add_metaclass(ReloadingMetaclass)

utool.util_class.reloading_meta_metaclass_factory(BASE_TYPE=<type 'type'>)[source]

hack for pyqt

utool.util_class.remove_private_obfuscation(self)[source]

removes the python obfuscation of class privates so they can be executed as they appear in class source. Useful when playing with IPython.

utool.util_class.test_reloading_metaclass()[source]

CommandLine:

python -m utool.util_class --test-test_reloading_metaclass

References

http://stackoverflow.com/questions/8122734/pythons-imp-reload-function-is-not-working

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_class import *  # NOQA
>>> result = test_reloading_metaclass()
>>> print(result)

utool.util_config module

util_config

TODO: FINISH ME AND ALLOW FOR CUSTOM SETTINGS BASED OFF OF A USER PROFILE

utool.util_config.get_default_global_config()[source]
utool.util_config.get_default_repo_config()[source]

import utool

utool.util_config.read_repo_config()[source]
utool.util_config.write_default_repo_config()[source]

utool.util_const module

class utool.util_const.ClassNoParam[source]

Bases: object

# class used in place of None when None might be a valid value # probably should just make None not a valid value

utool.util_cplat module

cross platform utilities

utool.util_cplat.assert_installed_debian(pkgname)[source]
utool.util_cplat.change_term_title(title)[source]

only works on unix systems only tested on Ubuntu GNOME changes text on terminal title for identifying debugging tasks.

The title will remain until python exists

Parameters:title (str) –

CommandLine:

python -m utool.util_cplat --test-change_term_title

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_cplat import *  # NOQA
>>> # build test data
>>> title = 'change title test'
>>> # execute function
>>> result = change_term_title(title)
>>> # verify results
>>> print(result)
utool.util_cplat.check_installed_debian(pkgname)[source]

References

http://www.cyberciti.biz/faq/find-out-if-package-is-installed-in-linux/

utool.util_cplat.chmod(fpath, option)[source]
utool.util_cplat.chmod_add_executable(fpath, group=True, user=True)[source]

References

http://stackoverflow.com/questions/15607903/python-module-os-chmodfile-664-does-not-change-the-permission-to-rw-rw-r-bu http://www.tutorialspoint.com/python/os_chmod.htm https://en.wikipedia.org/wiki/Chmod

utool.util_cplat.cmd(*args, **kwargs)[source]

A really roundabout way to issue a system call

# FIXME: This function needs some work # It should work without a hitch on windows or unix. # It should be able to spit out stdout in realtime. # Should be able to configure detatchment, shell, and sudo.

FIXME:
on a mac ut.cmd(‘/Users/joncrall/Library/Application Support/ibeis/tomcat/bin/shutdown.sh’) will fail due to spaces
Kwargs:
quiet (bool) : silence (bool) : verbose (bool) : detatch (bool) : shell (bool) : sudo (bool) : pad_stdout (bool) : dryrun (bool) :
Returns:(None, None, None)
Return type:tuple

CommandLine:

python -m utool.util_cplat --test-cmd
python -m utool.util_cplat --test-cmd:0
python -m utool.util_cplat --test-cmd:1
python -m utool.util_cplat --test-cmd:2
python -m utool.util_cplat --test-cmd:1 --test-sudo
python -m utool.util_cplat --test-cmd:2 --test-sudo
Example0:
>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> (out, err, ret) = ut.cmd('echo', 'hello world')
>>> result = ut.list_str(list(zip(('out', 'err', 'ret'), (out, err, ret))), nobraces=True)
>>> print(result)
('out', 'hello world\n'),
('err', None),
('ret', 0),
Example1:
>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> target = ut.codeblock(
...      r'''
         ('out', 'hello world\n'),
         ('err', None),
         ('ret', 0),
         ''')
>>> varydict = {
...    'shell': [True, False],
...    'detatch': [False],
...    'sudo': [True, False] if ut.get_argflag('--test-sudo') else [False],
...    'args': ['echo hello world', ('echo', 'hello world')],
... }
>>> for count, kw in enumerate(ut.all_dict_combinations(varydict), start=1):
>>>     print('+ --- TEST CMD %d ---' % (count,))
>>>     print('testing cmd with params ' + ut.dict_str(kw))
>>>     args = kw.pop('args')
>>>     restup = ut.cmd(args, pad_stdout=False, **kw)
>>>     tupfields = ('out', 'err', 'ret')
>>>     output = ut.list_str(list(zip(tupfields, restup)), nobraces=True)
>>>     ut.assert_eq(output, target)
>>>     print('L ___ TEST CMD %d ___\n' % (count,))
Example2:
>>> # UNSTABLE_DOCTEST
>>> # ping is not as universal of a command as I thought
>>> from utool.util_cplat import *  # NOQA
>>> import utool as ut
>>> varydict = {
...    'shell': [True, False],
...    'detatch': [True],
...    'args': ['ping localhost', ('ping', 'localhost')],
... }
>>> proc_list = []
>>> for count, kw in enumerate(ut.all_dict_combinations(varydict), start=1):
>>>     print('+ --- TEST CMD %d ---' % (count,))
>>>     print('testing cmd with params ' + ut.dict_str(kw))
>>>     args = kw.pop('args')
>>>     restup = ut.cmd(args, pad_stdout=False, **kw)
>>>     out, err, proc = restup
>>>     proc_list.append(proc)
>>>     print(proc)
>>>     print(proc)
>>>     print(proc.poll())
>>>     print('L ___ TEST CMD %d ___\n' % (count,))
utool.util_cplat.editfile(fpath)[source]

Runs gvim

utool.util_cplat.ensure_app_resource_dir(*args, **kwargs)[source]
utool.util_cplat.get_computer_name()[source]

Returns machine name

utool.util_cplat.get_dir_diskspaces(dir_)[source]
utool.util_cplat.get_disk_space(start_path=u'.')[source]

References

http://stackoverflow.com/questions/1392413/calculating-a-directory-size-using-python

utool.util_cplat.get_dynamic_lib_globstrs()[source]
utool.util_cplat.get_dynlib_dependencies(lib_path)[source]

Executes tools for inspecting dynamic library dependencies depending on the current platform.

utool.util_cplat.get_dynlib_exports(lib_path)[source]

Executes tools for inspecting dynamic library dependencies depending on the current platform. Returns the names of callable functions.

Parameters:lib_path (str) –
Returns:depend_out
Return type:str

CommandLine:

python -m utool.util_cplat --test-get_dynlib_exports

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_cplat import *  # NOQA
>>> lib_path = '/home/joncrall/venv/local/lib/python2.7/site-packages/pyflann/lib/libflann.so'
>>> depend_out = get_dynlib_exports(lib_path)
>>> result = ('depend_out = %s' % (str(depend_out),))
>>> print(result)
utool.util_cplat.get_file_info(fpath)[source]
utool.util_cplat.get_file_nBytes(fpath)[source]
utool.util_cplat.get_file_nBytes_str(fpath)[source]
utool.util_cplat.get_flops()[source]

# DOESNT WORK

utool.util_cplat.get_free_diskbytes(dir_)[source]
Parameters:dir (str) –
Returns:bytes_ folder/drive free space (in bytes)
Return type:int
References::
http://stackoverflow.com/questions/51658/cross-platform-space-remaining-on-volume-using-python http://linux.die.net/man/2/statvfs

CommandLine:

python -m utool.util_cplat --exec-get_free_diskbytes
python -m utool.util_cplat --exec-get_free_diskbytes --dir /media/raid
python -m utool.util_cplat --exec-get_free_diskbytes --dir E:

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_cplat import *  # NOQA
>>> import utool as ut
>>> dir_ = ut.get_argval('--dir', type_=str, default=ut.truepath('~'))
>>> bytes_ = get_free_diskbytes(dir_)
>>> result = ('bytes_ = %s' % (str(bytes_),))
>>> print(result)
>>> print('Unused space in %r = %r' % (dir_, ut.byte_str2(bytes_)))
>>> print('Total space in %r = %r' % (dir_, ut.byte_str2(get_total_diskbytes(dir_))))
utool.util_cplat.get_install_dirs()[source]
utool.util_cplat.get_lib_ext()[source]
utool.util_cplat.get_path_dirs()[source]

returns a list of directories in the PATH system variable

Returns:pathdirs
Return type:list

CommandLine:

python -m utool.util_cplat --exec-get_path_dirs

Example

>>> # SCRIPT
>>> from utool.util_cplat import *  # NOQA
>>> import utool as ut
>>> pathdirs = get_path_dirs()
>>> result = ('pathdirs = %s' % (ut.list_str(pathdirs),))
>>> print(result)
utool.util_cplat.get_pylib_ext()[source]
utool.util_cplat.get_python_dynlib()[source]

python -c “import utool; print(utool.get_python_dynlib())”

get_python_dynlib

Returns:dynlib
Return type:?

Example

>>> # DOCTEST_DISABLE
>>> from utool.util_cplat import *  # NOQA
>>> dynlib = get_python_dynlib()
>>> print(dynlib)
/usr/lib/x86_64-linux-gnu/libpython2.7.so
utool.util_cplat.get_system_python_library()[source]

FIXME; hacky way of finding python library. Not cross platform yet.

utool.util_cplat.get_total_diskbytes(dir_)[source]
utool.util_cplat.get_user_name()[source]

Returns user homefolder name

utool.util_cplat.geteditor()[source]
utool.util_cplat.getroot()[source]
utool.util_cplat.in_pyinstaller_package()[source]

References

http://stackoverflow.com/questions/22472124/what-is-sys-meipass-in-python http://stackoverflow.com/questions/7674790/bundling-data-files-with-pyinstaller-onefile

utool.util_cplat.ipython_paste(*args, **kwargs)[source]

pastes for me FIXME: make something like this work on unix and windows

utool.util_cplat.is64bit_python()[source]
Returns:True if running 64 bit python and False if running on 32 bit
utool.util_cplat.is_file_executable(fpath)[source]
utool.util_cplat.is_file_writable(fpath)[source]
utool.util_cplat.ls_libs(dpath)[source]
utool.util_cplat.pip_install(package)[source]

References

http://stackoverflow.com/questions/15974100/ipython-install-new-modules

utool.util_cplat.print_dir_diskspace(dir_)[source]
utool.util_cplat.print_path(sort=True)[source]
utool.util_cplat.print_system_users()[source]

prints users on the system

On unix looks for /bin/bash users in /etc/passwd

CommandLine:

python -m utool.util_cplat --test-print_system_users

Example

>>> # SCRIPT
>>> from utool.util_cplat import *  # NOQA
>>> result = print_system_users()
>>> print(result)
utool.util_cplat.python_executable(check=True, short=False)[source]
Parameters:short (bool) – (default = False)
Returns:
Return type:str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_cplat import *  # NOQA
>>> short = False
>>> result = python_executable(short)
>>> print(result)
utool.util_cplat.quote_single_command(cmdstr)[source]
utool.util_cplat.run_realtime_process(exe, shell=False)[source]
utool.util_cplat.search_env_paths(fname, key_list=None, verbose=None)[source]

Searches your PATH to see if fname exists

Parameters:fname (str) – file name to search for

CommandLine:

python -m utool.util_cplat --test-search_env_paths

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_cplat import *  # NOQA
>>> # build test data
>>> fname = 'opencv2/highgui/libopencv_highgui.so'
>>> # execute function
>>> result = search_env_paths(fname)
>>> # verify results
>>> print(result)
utool.util_cplat.send_keyboard_input(text=None, key_list=None)[source]
Parameters:
  • text (None) –
  • key_list (list) –

References

http://stackoverflow.com/questions/14788036/python-win32api-sendmesage http://www.pinvoke.net/default.aspx/user32.sendinput

CommandLine:

python -m utool.util_cplat --test-send_keyboard_input

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_cplat import *  # NOQA
>>> # build test data
>>> text = '%paste'
>>> # execute function
>>> result = send_keyboard_input('%paste')
>>> # verify results
>>> print(result)
utool.util_cplat.set_process_title(title)[source]
utool.util_cplat.shell(*args, **kwargs)[source]

Dangerous. Take out of production code

utool.util_cplat.spawn_delayed_ipython_paste()[source]
utool.util_cplat.startfile(fpath, detatch=True, quote=False, verbose=False, quiet=True)[source]

Uses default program defined by the system to open a file.

References

http://stackoverflow.com/questions/2692873/quote-posix-shell-special-characters-in-python-output

utool.util_cplat.unload_module(modname)[source]

WARNING POTENTIALLY DANGEROUS AND MAY NOT WORK

References

http://stackoverflow.com/questions/437589/how-do-i-unload-reload-a-python-module

CommandLine:

python -m utool.util_cplat --test-unload_module

Example

>>> import sys, gc  # NOQA
>>> import pyhesaff
>>> import utool as ut
>>> modname = 'pyhesaff'
>>> print('%s refcount=%r' % (modname, sys.getrefcount(pyhesaff),))
>>> #referrer_list = gc.get_referrers(sys.modules[modname])
>>> #print('referrer_list = %s' % (ut.list_str(referrer_list),))
>>> ut.unload_module(modname)
>>> assert pyhesaff is None
utool.util_cplat.vd(dname=None, verbose=True)

view directory

Parameters:
  • dname (str) – directory name
  • verbose (bool) –

CommandLine:

python -m utool.util_cplat --test-view_directory

Example

>>> # DOCTEST_DISABLE
>>> from utool.util_cplat import *  # NOQA
>>> import utool as ut
>>> dname = ut.truepath('~')
>>> verbose = True
>>> view_directory(dname, verbose)
utool.util_cplat.view_directory(dname=None, verbose=True)[source]

view directory

Parameters:
  • dname (str) – directory name
  • verbose (bool) –

CommandLine:

python -m utool.util_cplat --test-view_directory

Example

>>> # DOCTEST_DISABLE
>>> from utool.util_cplat import *  # NOQA
>>> import utool as ut
>>> dname = ut.truepath('~')
>>> verbose = True
>>> view_directory(dname, verbose)

utool.util_csv module

utool.util_csv.make_csv_table(column_list=[], column_lbls=None, header=u'', column_type=None, row_lbls=None, transpose=False, precision=2, use_lbl_width=True, comma_repl=u'<comma>', raw=False)[source]

Creates a csv table with aligned columns

make_csv_table

Parameters:
  • column_list (list) –
  • column_lbls (None) –
  • header (str) –
  • column_type (None) –
  • row_lbls (None) –
  • transpose (bool) –
Returns:

csv_text
Return type:

str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_csv import *  # NOQA
>>> column_list = [[1, 2, 3], ['A', 'B', 'C']]
>>> column_lbls = ['num', 'alpha']
>>> header = '# Test CSV'
>>> column_type = (int, str)
>>> row_lbls = None
>>> transpose = False
>>> csv_text = make_csv_table(column_list, column_lbls, header, column_type, row_lbls, transpose)
>>> result = csv_text
>>> print(result)
# Test CSV
# num_rows=3
#   num,  alpha
      1,      A
      2,      B
      3,      C
utool.util_csv.numpy_to_csv(arr, col_lbls=None, header=u'', col_type=None)[source]
utool.util_csv.read_csv(fpath)[source]

reads csv in unicode

utool.util_dbg module

class utool.util_dbg.EmbedOnException[source]

Bases: object

Context manager which embeds in ipython if an exception is thrown

utool.util_dbg.all_rrr()[source]
utool.util_dbg.breakpoint(*tags)[source]
utool.util_dbg.debug_exception(func)[source]
utool.util_dbg.debug_hstack(stacktup)[source]
utool.util_dbg.debug_list(list_)[source]
utool.util_dbg.debug_npstack(stacktup)[source]
utool.util_dbg.debug_vstack(stacktup)[source]
utool.util_dbg.dict_dbgstr(dict_name, locals_=None)[source]
utool.util_dbg.embed(parent_locals=None, parent_globals=None, exec_lines=None, remove_pyqt_hook=True, N=0)[source]

Starts interactive session. Similar to keyboard command in matlab. Wrapper around IPython.embed

Notes

#https://github.com/ipython/ipython/wiki/Cookbook%3a-Updating-code-for-use-with-IPython-0.11-and-later

import IPython x = 3 IPython.embed() c = IPython.Config() c.InteractiveShellApp.exec_lines = [

‘%pylab qt4’, “print ‘System Ready!’”,

]

def foo():
return x + 3

a = 3 def bar():

return a + 3

bar() #NameError: global name ‘a’ is not defined

from IPython.terminal.ipapp import TerminalIPythonApp x = 3 app = TerminalIPythonApp.instance() app.initialize(argv=[]) # argv=[] instructs IPython to ignore sys.argv app.start()

Parameters:
  • parent_locals (None) –
  • parent_globals (None) –
  • exec_lines (None) –
  • remove_pyqt_hook (bool) –
  • N (int) –

CommandLine:

python -m utool.util_dbg --test-embed

References

http://stackoverflow.com/questions/27911570/can-you-specify-a-command-to-run-after-you-embed-into-ipython/27914204#27914204 http://stackoverflow.com/questions/15167200/how-do-i-embed-an-ipython-interpreter-into-an-application-running-in-an-ipython

TODO:
try:
get_ipython
except NameError:
banner=exit_msg=’‘
else:
banner = ‘* Nested interpreter *‘ exit_msg = ‘* Back in main IPython *‘

# First import the embed function from IPython.frontend.terminal.embed import InteractiveShellEmbed # Now create the IPython shell instance. Put ipshell() anywhere in your code # where you want it to open. ipshell = InteractiveShellEmbed(banner1=banner, exit_msg=exit_msg) #Then use ipshell() whenever you want to be dropped into an IPython shell. This #will allow you to embed (and even nest) IPython interpreters in your code and #inspect objects or the state of the program.

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dbg import *  # NOQA
>>> # build test data
>>> parent_locals = None
>>> parent_globals = None
>>> exec_lines = None
>>> remove_pyqt_hook = True
>>> N = 0
>>> # execute function
>>> result = embed(parent_locals, parent_globals, exec_lines, remove_pyqt_hook, N)
>>> # verify results
>>> print(result)
utool.util_dbg.embed2(**kwargs)[source]

Modified from IPython.terminal.embed.embed so I can mess with stack_depth

utool.util_dbg.execstr_attr_list(obj_name, attr_list=None)[source]
utool.util_dbg.execstr_dict(dict_, local_name=None, exclude_list=None, explicit=False)[source]

returns execable python code that declares variables using keys and values

execstr_dict

Parameters:
  • dict (dict) –
  • local_name (str) – optional: local name of dictionary. Specifying this is much safer
  • exclude_list (list) –
Returns:

execstr — the executable string that will put keys from dict into local vars
Return type:

str

CommandLine:

python -m utool.util_dbg --test-execstr_dict

Example

>>> # UNSTABLE_DOCTEST
>>> from utool.util_dbg import *  # NOQA
>>> my_dictionary = {'a': True, 'b':False}
>>> execstr = execstr_dict(my_dictionary)
>>> exec(execstr)
>>> assert 'a' in vars() and 'b' in vars(), 'execstr failed'
>>> assert b is False and a is True, 'execstr failed'
>>> result = execstr
>>> print(result)
a = my_dictionary['a']
b = my_dictionary['b']

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dbg import *  # NOQA
>>> import utool as ut
>>> my_dictionary = {'a': True, 'b':False}
>>> execstr = execstr_dict(my_dictionary)
>>> locals_ = locals()
>>> exec(execstr, locals_)
>>> a, b = ut.dict_take(locals_, ['a', 'b'])
>>> assert 'a' in locals_ and 'b' in locals_, 'execstr failed'
>>> assert b is False and a is True, 'execstr failed'
>>> result = execstr
>>> print(result)
a = my_dictionary['a']
b = my_dictionary['b']

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dbg import *  # NOQA
>>> import utool as ut
>>> my_dictionary = {'a': True, 'b':False}
>>> execstr = execstr_dict(my_dictionary, explicit=True)
>>> result = execstr
>>> print(result)
a = True
b = False
utool.util_dbg.execstr_func(func)[source]
utool.util_dbg.execstr_parent_locals()[source]
utool.util_dbg.explore_module(module_, seen=None, maxdepth=2, nonmodules=False)[source]
utool.util_dbg.explore_stack()[source]
utool.util_dbg.fix_embed_globals()[source]

HACK adds current locals() to globals(). Can be dangerous.

utool.util_dbg.fmtlocals(string)
utool.util_dbg.formatex(ex, msg=u'[!?] Caught exception', prefix=None, key_list=[], locals_=None, iswarning=False, tb=False, N=0, keys=None, colored=None)[source]

Formats an exception with relevant info

Parameters:
  • ex (Exception) – exception to print
  • msg (unicode) – a message to display to the user (default = u’[!?] Caught exception’)
  • keys (None) – a list of strings denoting variables or expressions of interest (default = [])
  • iswarning (bool) – prints as a warning rather than an error if True (default = False)
  • tb (bool) – if True prints the traceback in the error message (default = False)
  • prefix (None) – (default = None)
  • locals (None) – (default = None)
  • N (int) – (default = 0)
  • colored (None) – (default = None)
  • key_list (list) – DEPRICATED use keys
Returns:

formated exception
Return type:

str

CommandLine:

python -m utool.util_dbg --exec-formatex

Example

>>> # ENABLE_DOCTET
>>> from utool.util_dbg import *  # NOQA
>>> import utool as ut
>>> msg = 'Testing Exception'
>>> prefix = '[test]'
>>> key_list = ['N', 'foo', 'tb']
>>> locals_ = None
>>> iswarning = True
>>> keys = None
>>> colored = None
>>> def failfunc():
>>>     tb = True
>>>     N = 0
>>>     try:
>>>         raise Exception('test exception. This is not an error')
>>>     except Exception as ex:
>>>         result = formatex(ex, msg, prefix, key_list, locals_,
>>>                           iswarning, tb, N, keys, colored)
>>>         return result
>>> result = failfunc().replace('\n\n', '')
>>> print(result)
<!!! WARNING !!!>
Traceback (most recent call last):
  File "<string>", line 15, in failfunc
Exception: test exception. This is not an error[test] Testing Exception
<type 'exceptions.Exception'>: test exception. This is not an error
[test] N = 0
[test] !!! foo not populated!
[test] tb = True
</!!! WARNING !!!>
utool.util_dbg.get_caller_lineno(N=0, strict=True)[source]
utool.util_dbg.get_caller_locals(N=0)[source]

returns the locals of the function that called you alias for get_parent_locals

Parameters:N (int) – (defaults to 0) number of levels up in the stack
Returns:locals_
Return type:dict
utool.util_dbg.get_caller_modname(N=0, allowmain=True)[source]
utool.util_dbg.get_caller_name(N=0, allow_genexpr=True)[source]

get the name of the function that called you

Parameters:N (int) – (defaults to 0) number of levels up in the stack
Returns:a function name
Return type:str
utool.util_dbg.get_caller_prefix(N=0, aserror=False)[source]
utool.util_dbg.get_caller_stack_frame(N=0)[source]
utool.util_dbg.get_current_stack_depth()[source]
utool.util_dbg.get_localvar_from_stack(varname)

Finds a local varable somewhere in the stack and returns the value

Parameters:varname (str) – variable name
Returns:None if varname is not found else its value
utool.util_dbg.get_parent_frame(N=0)[source]
utool.util_dbg.get_parent_globals(N=0)[source]
utool.util_dbg.get_parent_locals(N=0)[source]

returns the locals of the function that called you

Parameters:N (int) – (defaults to 0) number of levels up in the stack
Returns:locals_
Return type:dict
utool.util_dbg.get_reprs(*args, **kwargs)[source]
utool.util_dbg.get_stack_frame(N=0, strict=True)[source]
utool.util_dbg.get_var_from_stack(varname, verbose=True)

Finds a varable (local or global) somewhere in the stack and returns the value

Parameters:varname (str) – variable name
Returns:None if varname is not found else its value
utool.util_dbg.get_varname_from_locals(val, locals_, default=u'varname-not-found', strict=False, cmpfunc_=<built-in function is_>)[source]

Finds the string name which has where locals_[name] is val

Check the varname is in the parent namespace This will only work with objects not primatives

Parameters:
  • () (val) – some value
  • locals (dict) – local dictionary to search
  • default (str) –
  • strict (bool) –
Returns:

the varname which is Val (if it exists)
Return type:

str
utool.util_dbg.get_varname_from_stack(var, N=0, **kwargs)[source]
utool.util_dbg.get_varstr(val, pad_stdout=True, locals_=None)[source]
utool.util_dbg.get_varval_from_locals(key, locals_, strict=False)[source]

Returns a variable value from locals. Different from locals()[‘varname’] because get_varval_from_locals(‘varname.attribute’, locals()) is allowed

utool.util_dbg.haveIPython()[source]

Tests if IPython is available

utool.util_dbg.import_testdata()[source]
utool.util_dbg.inIPython()[source]

Tests if running in IPython

utool.util_dbg.in_jupyter_notebook()[source]

http://stackoverflow.com/questions/15411967/how-can-i-check-if-code-is-executed-in-the-ipython-notebook

utool.util_dbg.ipython_execstr()[source]
utool.util_dbg.is_valid_varname(varname)[source]

Checks syntax and validity of a variable name

utool.util_dbg.keys_dbgstr(dict_name, locals_=None)[source]
utool.util_dbg.len_dbgstr(lenable_name, locals_=None)[source]
utool.util_dbg.list_dbgstr(list_name, trunc=2)[source]
utool.util_dbg.load_testdata(*args, **kwargs)[source]

tries to load previously cached testdata

utool.util_dbg.module_functions(module)[source]
utool.util_dbg.parse_locals_keylist(locals_, key_list, strlist_=None, prefix=u'')[source]

For each key in keylist, puts its value in locals into a stringlist

Parameters:
  • locals –
  • key_list (list) –
  • strlist (list) – (default = None)
  • prefix (unicode) – (default = u’‘)
Returns:

strlist_
Return type:

list

CommandLine:

python -m utool.util_dbg --exec-parse_locals_keylist

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dbg import *  # NOQA
>>> import utool as ut
>>> locals_ = {'foo': [1,2,3], 'bar': 'spam', 'eggs': 4, 'num': 5}
>>> key_list = [(len, 'foo'), 'bar.lower.__name__', 'eggs', 'num', 'other']
>>> strlist_ = None
>>> prefix = u''
>>> strlist_ = parse_locals_keylist(locals_, key_list, strlist_, prefix)
>>> result = ('strlist_ = %s' % (ut.repr2(strlist_, nl=True),))
>>> print(result)
strlist_ = [
    ' len(foo) = 3',
    " bar.lower.__name__ = 'lower'",
    ' eggs = 4',
    ' num = 5',
    ' other = NameError',
]
utool.util_dbg.print_frame(frame)[source]
utool.util_dbg.print_keys(key_list, locals_=None)[source]
utool.util_dbg.print_traceback(with_colors=True)[source]

prints current stack

utool.util_dbg.print_varlen(name_, locals_=None)[source]
utool.util_dbg.printex(ex, msg=u'[!?] Caught exception', prefix=None, key_list=[], locals_=None, iswarning=False, tb=False, pad_stdout=True, N=0, use_stdout=False, reraise=False, msg_=None, keys=None, colored=None)[source]

Prints (and/or logs) an exception with relevant info

Parameters:
  • ex (Exception) – exception to print
  • msg (None) – a message to display to the user
  • keys (None) – a list of strings denoting variables or expressions of interest
  • iswarning (bool) – prints as a warning rather than an error if True (defaults to False)
  • tb (bool) – if True prints the traceback in the error message
  • pad_stdout (bool) – separate the error message from the rest of stdout with newlines
  • prefix (None) –
  • locals (None) –
  • N (int) –
  • use_stdout (bool) –
  • reraise (bool) –
  • msg –
  • key_list (list) – DEPRICATED use keys
Returns:

None
utool.util_dbg.printvar(locals_, varname, attr=u'.shape', typepad=0)[source]
utool.util_dbg.printvar2(varstr, attr=u'', typepad=0)[source]
utool.util_dbg.public_attributes(input)[source]
utool.util_dbg.qflag(num=None, embed_=True)[source]
utool.util_dbg.quasiquote(string)[source]
utool.util_dbg.quit(num=None, embed_=False)[source]
utool.util_dbg.quitflag(num=None, embed_=False, parent_locals=None, parent_globals=None)[source]
utool.util_dbg.save_testdata(*args, **kwargs)[source]

caches testdata

utool.util_dbg.search_stack_for_localvar(varname)[source]

Finds a local varable somewhere in the stack and returns the value

Parameters:varname (str) – variable name
Returns:None if varname is not found else its value
utool.util_dbg.search_stack_for_var(varname, verbose=True)[source]

Finds a varable (local or global) somewhere in the stack and returns the value

Parameters:varname (str) – variable name
Returns:None if varname is not found else its value
utool.util_dbg.super_print(val, locals_=None)[source]

utool.util_decor module

utool.util_decor.accepts_numpy(func)[source]

Allows the first input to be a numpy array and get result in numpy form

utool.util_decor.accepts_scalar_input(func)[source]

DEPRICATE in favor of accepts_scalar_input2 only accepts one input as vector

accepts_scalar_input is a decorator which expects to be used on class methods. It lets the user pass either a vector or a scalar to a function, as long as the function treats everything like a vector. Input and output is sanitized to the user expected format on return.

Parameters:func (func) –
Returns:wrp_asi
Return type:func

CommandLine:

python -m utool.util_decor --test-accepts_scalar_input

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_decor import *  # NOQA
>>> @accepts_scalar_input
... def foobar(self, list_):
...     return [x + 1 for x in list_]
>>> self = None  # dummy self because this decorator is for classes
>>> assert 2 == foobar(self, 1)
>>> assert [2, 3] == foobar(self, [1, 2])
utool.util_decor.accepts_scalar_input2(argx_list=[0], outer_wrapper=True)[source]

FIXME: change to better name. Complete implementation.

used in IBEIS setters

accepts_scalar_input2 is a decorator which expects to be used on class methods. It lets the user pass either a vector or a scalar to a function, as long as the function treats everything like a vector. Input and output is sanitized to the user expected format on return.

Parameters:argx_list (list) – indexes of args that could be passed in as scalars to code that operates on lists. Ensures that decorated function gets the argument as an iterable.
utool.util_decor.accepts_scalar_input_vector_output(func)[source]

DEPRICATE IN FAVOR OF accepts_scalar_input2

accepts_scalar_input_vector_output

Notes

Input: Excpeted Output 1to1 Expected Output 1toM
scalar : 1 x [X] n element list : [1, 2, 3] [x, y, z] [[X], [Y], [Z]] 1 element list : [1] [x] [[X]] 0 element list : [] [] []

There seems to be no real issue here, I be the thing that tripped me up was when using sql and getting multiple columns that returned the values inside of the N-tuple whereas when you get one column you get one element inside of a 1-tuple, no that still makes sense. There was something where when you couln’t unpack it becuase it was already empty...

utool.util_decor.apply_docstr(docstr_func)[source]

Changes docstr of one functio to that of another

utool.util_decor.dummy_args_decor(*args, **kwargs)[source]
utool.util_decor.getter_1to1(func)

DEPRICATE in favor of accepts_scalar_input2 only accepts one input as vector

accepts_scalar_input is a decorator which expects to be used on class methods. It lets the user pass either a vector or a scalar to a function, as long as the function treats everything like a vector. Input and output is sanitized to the user expected format on return.

Parameters:func (func) –
Returns:wrp_asi
Return type:func

CommandLine:

python -m utool.util_decor --test-accepts_scalar_input

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_decor import *  # NOQA
>>> @accepts_scalar_input
... def foobar(self, list_):
...     return [x + 1 for x in list_]
>>> self = None  # dummy self because this decorator is for classes
>>> assert 2 == foobar(self, 1)
>>> assert [2, 3] == foobar(self, [1, 2])
utool.util_decor.getter_1toM(func)

DEPRICATE IN FAVOR OF accepts_scalar_input2

accepts_scalar_input_vector_output

Notes

Input: Excpeted Output 1to1 Expected Output 1toM
scalar : 1 x [X] n element list : [1, 2, 3] [x, y, z] [[X], [Y], [Z]] 1 element list : [1] [x] [[X]] 0 element list : [] [] []

There seems to be no real issue here, I be the thing that tripped me up was when using sql and getting multiple columns that returned the values inside of the N-tuple whereas when you get one column you get one element inside of a 1-tuple, no that still makes sense. There was something where when you couln’t unpack it becuase it was already empty...

utool.util_decor.indent_func(input_)[source]

Takes either no arguments or an alias label

utool.util_decor.interested(func)[source]
utool.util_decor.lazyfunc(func)[source]

Returns a memcached version of a function

utool.util_decor.memoize(func)[source]

simple memoization decorator

References

https://wiki.python.org/moin/PythonDecoratorLibrary#Memoize

Parameters:func (function) – live python function
Returns:
Return type:func

CommandLine:

python -m utool.util_decor --exec-memoize --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_decor import *  # NOQA
>>> import utool as ut
>>> closure = {'a': 'b', 'c': 'd'}
>>> incr = [0]
>>> def foo(key):
>>>    value = closure[key]
>>>    incr[0] += 1
>>>    return value
>>> foo_memo = memoize(foo)
>>> assert foo('a') == 'b' and foo('c') == 'd'
>>> assert incr[0] == 2
>>> print('Call memoized version')
>>> assert foo_memo('a') == 'b' and foo_memo('c') == 'd'
>>> assert incr[0] == 4
>>> assert foo_memo('a') == 'b' and foo_memo('c') == 'd'
>>> print('Counter should no longer increase')
>>> assert incr[0] == 4
>>> print('Closure changes result without memoization')
>>> closure = {'a': 0, 'c': 1}
>>> assert foo('a') == 0 and foo('c') == 1
>>> assert incr[0] == 6
>>> assert foo_memo('a') == 'b' and foo_memo('c') == 'd'
utool.util_decor.memoize_nonzero(func)[source]

Memoization decorator for functions taking a nonzero number of arguments.

References

http://code.activestate.com/recipes/578231-fastest-memoization-decorator

utool.util_decor.memoize_single(func)[source]

Memoization decorator for a function taking a single argument

References

http://code.activestate.com/recipes/578231-fastest-memoization-decorator

utool.util_decor.memoize_zero(func)[source]

Memoization decorator for a function taking no arguments

utool.util_decor.on_exception_report_input(func_=None, force=False, keys=None)[source]

If an error is thrown in the scope of this function’s stack frame then the decorated function name and the arguments passed to it will be printed to the utool print function.

utool.util_decor.preserve_sig(wrapper, orig_func, force=False)[source]

Decorates a wrapper function.

It seems impossible to presever signatures in python 2 without eval (Maybe another option is to write to a temporary module?)

Parameters:
  • wrapper – the function wrapping orig_func to change the signature of
  • orig_func – the original function to take the signature from

References

http://emptysqua.re/blog/copying-a-python-functions-signature/ https://code.google.com/p/micheles/source/browse/decorator/src/decorator.py

TODO:
checkout funcsigs https://funcsigs.readthedocs.org/en/latest/

CommandLine:

python -m utool.util_decor --test-preserve_sig

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> #ut.rrrr(False)
>>> def myfunction(self, listinput_, arg1, *args, **kwargs):
>>>     " just a test function "
>>>     return [x + 1 for x in listinput_]
>>> #orig_func = ut.take
>>> orig_func = myfunction
>>> wrapper = ut.accepts_scalar_input2([0])(orig_func)
>>> _wrp_preserve1 = ut.preserve_sig(wrapper, orig_func, True)
>>> _wrp_preserve2 = ut.preserve_sig(wrapper, orig_func, False)
>>> print('_wrp_preserve2 = %r' % (_wrp_preserve1,))
>>> print('_wrp_preserve2 = %r' % (_wrp_preserve2,))
>>> print('source _wrp_preserve1 = %s' % (ut.get_func_sourcecode(_wrp_preserve1),))
>>> print('source _wrp_preserve2 = %s' % (ut.get_func_sourcecode(_wrp_preserve2)),)
>>> result = str(_wrp_preserve1)
>>> print(result)
utool.util_decor.show_return_value(func)[source]
utool.util_decor.test_ignore_exec_traceback()[source]

CommandLine:

python -m utool.util_decor --test-test_ignore_exec_traceback

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_decor import *  # NOQA
>>> result = test_ignore_exec_traceback()
>>> print(result)
utool.util_decor.time_func(func)[source]
utool.util_decor.tracefunc(func)[source]
utool.util_decor.tracefunc_xml(func)[source]

Causes output of function to be printed in an XML style block

utool.util_depricated module

utool.util_depricated.cartesian(arrays, out=None)[source]

Generate a cartesian product of input arrays.

Args:
arrays (list of array-like): 1-D arrays to form the cartesian product of out (ndarray): Outvar which is modified in place if specified
Returns:
out (ndarray): cartesian products formed of input arrays
2-D array of shape (M, len(arrays))
References:
gist.github.com/hernamesbarbara/68d073f551565de02ac5
Example:
>>> # ENABLE_DOCTEST
>>> from utool.util_depricated import *  # NOQA
>>> arrays = ([1, 2, 3], [4, 5], [6, 7])
>>> out = cartesian(arrays)
>>> result = repr(out.T)
array([[1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3],
       [4, 4, 5, 5, 4, 4, 5, 5, 4, 4, 5, 5],
       [6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7]])
Timeit:
>>> # Use itertools product instead
>>> setup = 'import utool as ut
‘ + ut.get_doctest_examples(ut.cartesian)[0][0]
>>> statements = [
>>>     'cartesian(arrays)',
>>>     'np.array(list(ut.iprod(*arrays)))',
>>> ]
>>> ut.timeit_compare(statements, setup=setup)
utool.util_depricated.find_std_inliers(data, m=2)[source]

utool.util_dev module

class utool.util_dev.AlignedListDictProxy(key2_idx, key_list, val_list)[source]

Bases: utool.util_dev.DictLike_old

simulates a dict when using parallel lists the point of this class is that when there are many instances of this class, then key2_idx can be shared between them. Ideally this class wont be used and will disappear when the parallel lists are being used properly.

iteritems()[source]
iterkeys()[source]
itervalues()[source]
pop(key)[source]
class utool.util_dev.ClassAttrDictProxy(obj, keys, attrs=None)[source]

Bases: utool.util_dev.DictLike_old

iterkeys()[source]
utool.util_dev.DEPRICATED(func)[source]

deprication decorator

class utool.util_dev.DictLike_old[source]

Bases: object

todo use util_dict.DictLike instead

copy()[source]
get(key, default=None)[source]
items()[source]
iteritems()[source]
itervalues()[source]
keys()[source]
values()[source]
class utool.util_dev.InteractiveIter(iiter, iterable=None, enabled=True, startx=0, default_action=u'next', custom_actions=[], wraparound=False, display_item=False, verbose=True)[source]

Bases: object

Choose next value interactively

iterable should be a list, not a generator. sorry

classmethod eventloop(custom_actions=[])[source]

For use outside of iteration wrapping. Makes an interactive event loop custom_actions should be specified in format [dispname, keys, desc, func]

handle_ans(iiter, ans_)[source]
prompt(iiter)[source]
class utool.util_dev.MemoryTracker(lbl=u'Memtrack Init', disable=None)[source]

Bases: object

A class for tracking memory usage. On initialization it logs the current available (free) memory. Calling the report method logs the current available memory as well as memory usage difference w.r.t the last report.

Example

>>> import utool
>>> import numpy as np
>>> memtrack = utool.MemoryTracker('[ENTRY]')
>>> memtrack.report('[BEFORE_CREATE]')
>>> arr = np.ones(128 * (2 ** 20), dtype=np.uint8)
>>> memtrack.report('[AFTER_CREATE]')
>>> memtrack.track_obj(arr, 'arr')
>>> memtrack.report_objs()
>>> memtrack.report_largest()
>>> del arr
>>> memtrack.report('[DELETE]')
#>>> memtrack.report_largest()
collect(*args, **kwargs)[source]
get_available_memory(*args, **kwargs)[source]
get_used_memory(*args, **kwargs)[source]
report(*args, **kwargs)[source]
report_largest(*args, **kwargs)[source]
report_obj(*args, **kwargs)[source]
report_objs(*args, **kwargs)[source]
report_type(class_, more=False)[source]
track_obj(*args, **kwargs)[source]
class utool.util_dev.NiceRepr[source]

Bases: object

base class that defines a nice representation and string func for a class given that the user implements __nice__

Rename to NiceObject?

utool.util_dev.are_you_sure(msg=u'')[source]

Prompts user to accept or checks command line for -y

Parameters:msg (str) –
Returns:accept or not
Return type:bool
utool.util_dev.argparse_funckw(func, defaults={}, **kwargs)[source]

allows kwargs to be specified on the commandline from testfuncs

Parameters:func (function) –
Kwargs:
lbl, verbose, only_specified, force_keys, type_hint, alias_dict
Returns:funckw
Return type:dict

CommandLine:

python -m utool.util_dev --exec-argparse_funckw

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> func = argparse_funckw
>>> funckw = argparse_funckw(func)
>>> result = ('funckw = %s' % (str(funckw),))
>>> print(result)
utool.util_dev.autopep8_diff(fpath)[source]
Parameters:fpath (str) – file path string

CommandLine:

python -m utool.util_dev --test-autopep8_diff --fpath ingest_data.py

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> fpath = ut.get_argval('--fpath', type_=str, default='ingest_data.py')
>>> result = autopep8_diff(fpath)
>>> print(result)
utool.util_dev.compile_cython(fpath, clean=True)[source]

Compiles a cython pyx into a shared library

This seems broken compiles pyx -> pyd/dylib/so

Examples

REAL SETUP.PY OUTPUT cythoning vtool/linalg_cython.pyx to vtoollinalg_cython.c C:MinGWbingcc.exe -mdll -O -Wall ^ -IC:Python27Libsite-packagesnumpycoreinclude ^ -IC:Python27include -IC:Python27PC ^ -c vtoollinalg_cython.c ^ -o buildtemp.win32-2.7Releasevtoollinalg_cython.o

writing buildtemp.win32-2.7Releasevtoollinalg_cython.def

C:MinGWbingcc.exe -shared -s buildtemp.win32-2.7Releasevtoollinalg_cython.o buildtemp.win32-2.7Releasevtoollinalg_cython.def -LC:Python27libs -LC:Python27PCbuild -lpython27 -lmsvcr90 -o buildlib.win32-2.7vtoollinalg_cython.pyd

utool.util_dev.copy_text_to_clipboard(text)[source]

alias

utool.util_dev.dev_ipython_copypaster(func)[source]
utool.util_dev.disable_garbage_collection()[source]
utool.util_dev.enable_garbage_collection()[source]
utool.util_dev.ensure_pylab_qt4()[source]
utool.util_dev.ensure_str_list(input_)[source]
utool.util_dev.execstr_funckw(func)[source]

for doctests kwargs

SeeAlso:
ut.exec_func_src
utool.util_dev.extract_timeit_setup(func, doctest_number, sentinal)[source]
utool.util_dev.find_exe(name, path_hints=[], required=True)[source]
utool.util_dev.find_interesting_stats(stat_dict, col_lbls=None, lbl=None)[source]
utool.util_dev.garbage_collect()[source]
utool.util_dev.get_clipboard()[source]

References

http://stackoverflow.com/questions/11063458/python-script-to-copy-text-to-clipboard

utool.util_dev.get_cython_exe()[source]
utool.util_dev.get_dev_paste_code(func)[source]
utool.util_dev.get_jagged_stats(arr_list, **kwargs)[source]
Parameters:arr_list (list) –
Returns:stats_dict
Return type:dict

CommandLine:

python -m utool.util_dev --test-get_jagged_stats

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> kwargs = dict(use_nan=True)
>>> arr_list = [[1, 2, 3, 4], [3, 10], [np.nan, 3, 3, 3]]
>>> # execute function
>>> stats_dict = get_jagged_stats(arr_list, **kwargs)
>>> # verify results
>>> result = ut.align(str(ut.dict_str(stats_dict)), ':')
>>> print(result)
{
    'max'    : [4.0, 10.0, 3.0],
    'min'    : [1.0, 3.0, 3.0],
    'mean'   : [2.5, 6.5, 3.0],
    'std'    : [1.118034, 3.5, 0.0],
    'nMin'   : [1, 1, 3],
    'nMax'   : [1, 1, 3],
    'shape'  : ['(4,)', '(2,)', '(4,)'],
    'num_nan': [0, 0, 1],
}
utool.util_dev.get_nonconflicting_path(path_, dpath=None, offset=0)[source]
utool.util_dev.get_nonconflicting_path_old(base_fmtstr, dpath, offset=0)[source]

base_fmtstr must have a %d in it

utool.util_dev.get_nonconflicting_string(base_fmtstr, conflict_set, offset=0)[source]

gets a new string that wont conflict with something that already exists

Parameters:
  • base_fmtstr (str) –
  • conflict_set (set) –

CommandLine:

python -m utool.util_dev --test-get_nonconflicting_string

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> # build test data
>>> base_fmtstr = 'somestring%d'
>>> conflict_set = ['somestring0']
>>> # execute function
>>> result = get_nonconflicting_string(base_fmtstr, conflict_set)
>>> # verify results
>>> print(result)
somestring1
utool.util_dev.get_object_base()[source]
utool.util_dev.get_object_size(obj, fallback_type=None, follow_pointers=False, exclude_modules=True, listoverhead=False)[source]

CommandLine:

python -m utool.util_dev --test-get_object_size
python -m utool.util_dev --test-get_object_size:1

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool as ut
>>> obj = [np.empty(1, dtype=np.uint8) for _ in range(8)]
>>> nBytes = ut.get_object_size(obj)
>>> result = ('nBytes = %s' % (nBytes,))
>>> print(result)
nBytes = 8

Example

>>> # UNSTABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import ibeis
>>> import utool as ut
>>> species = ibeis.const.TEST_SPECIES.ZEB_PLAIN
>>> ibs = ibeis.opendb(defaultdb='testdb1')
>>> qaids = ibs.get_valid_aids(species=species)
>>> daids = ibs.get_valid_aids(species=species)
>>> qreq_ = ibs.new_query_request(qaids, daids, verbose=True)
>>> nBytes = ut.get_object_size(qreq_)
>>> result = (ut.byte_str2(nBytes))
>>> print('result = %r' % (result,))
utool.util_dev.get_object_size_str(obj, lbl=u'', unit=None)[source]

CommandLine:

python -m utool.util_dev --test-get_object_size_str --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool as ut
>>> obj = [np.empty((512), dtype=np.uint8) for _ in range(10)]
>>> nBytes = ut.get_object_size_str(obj)
>>> result = ('result = %s' % (ut.repr2(nBytes),))
>>> print(result)
result = '5.00 KB'
utool.util_dev.get_partial_func_name(func, precision=3)[source]
utool.util_dev.get_statdict(list_, axis=None, use_nan=False, use_sum=False, datacast=None, use_median=False)
TODO:
depricate datacast
Parameters:
  • list (listlike) – values to get statistics of
  • axis (int) – if list_ is ndarray then this specifies the axis
Returns:

stat_dict - dictionary of common numpy statistics (min, max, mean, std, nMin, nMax, shape)
Return type:

OrderedDict

CommandLine:

python -m utool.util_dev --test-get_stats
python -m utool.util_dev --test-get_stats:1
Examples0:
>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool
>>> axis = 0
>>> np.random.seed(0)
>>> list_ = np.random.rand(10, 2).astype(np.float32)
>>> stat_dict = get_stats(list_, axis, use_nan=False)
>>> result = str(utool.dict_str(stat_dict))
>>> print(result)
{
    'max': np.array([ 0.96366274,  0.92559665], dtype=np.float32),
    'min': np.array([ 0.0202184,  0.0871293], dtype=np.float32),
    'mean': np.array([ 0.52056623,  0.64254338], dtype=np.float32),
    'std': np.array([ 0.28543401,  0.25168759], dtype=np.float32),
    'nMin': np.array([1, 1], dtype=np.int32),
    'nMax': np.array([1, 1], dtype=np.int32),
    'shape': (10, 2),
}
Examples1:
>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool
>>> axis = 0
>>> rng = np.random.RandomState(0)
>>> list_ = rng.randint(0, 42, size=100).astype(np.float32)
>>> list_[4] = np.nan
>>> stat_dict = get_stats(list_, axis, use_nan=True)
>>> result = str(utool.dict_str(stat_dict))
>>> print(result)
{
    'max': 41.0,
    'min': 0.0,
    'mean': 20.0,
    'std': 13.177115,
    'nMin': 7,
    'nMax': 3,
    'shape': (100,),
    'num_nan': 1,
}
Examples1:
>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool
>>> axis = 0
>>> rng = np.random.RandomState(0)
>>> list_ = rng.randint(0, 42, size=100).astype(np.int32)
>>> stat_dict = get_stats(list_, axis, use_nan=True)
>>> result = str(utool.dict_str(stat_dict))
>>> print(result)
{
    'max': 41,
    'min': 0,
    'mean': 19.889999,
    'std': 13.156668,
    'nMin': 7,
    'nMax': 3,
    'shape': (100,),
    'num_nan': 0,
}
SeeAlso:
print_stats get_stats_str
utool.util_dev.get_stats(list_, axis=None, use_nan=False, use_sum=False, datacast=None, use_median=False)[source]
TODO:
depricate datacast
Parameters:
  • list (listlike) – values to get statistics of
  • axis (int) – if list_ is ndarray then this specifies the axis
Returns:

stat_dict - dictionary of common numpy statistics (min, max, mean, std, nMin, nMax, shape)
Return type:

OrderedDict

CommandLine:

python -m utool.util_dev --test-get_stats
python -m utool.util_dev --test-get_stats:1
Examples0:
>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool
>>> axis = 0
>>> np.random.seed(0)
>>> list_ = np.random.rand(10, 2).astype(np.float32)
>>> stat_dict = get_stats(list_, axis, use_nan=False)
>>> result = str(utool.dict_str(stat_dict))
>>> print(result)
{
    'max': np.array([ 0.96366274,  0.92559665], dtype=np.float32),
    'min': np.array([ 0.0202184,  0.0871293], dtype=np.float32),
    'mean': np.array([ 0.52056623,  0.64254338], dtype=np.float32),
    'std': np.array([ 0.28543401,  0.25168759], dtype=np.float32),
    'nMin': np.array([1, 1], dtype=np.int32),
    'nMax': np.array([1, 1], dtype=np.int32),
    'shape': (10, 2),
}
Examples1:
>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool
>>> axis = 0
>>> rng = np.random.RandomState(0)
>>> list_ = rng.randint(0, 42, size=100).astype(np.float32)
>>> list_[4] = np.nan
>>> stat_dict = get_stats(list_, axis, use_nan=True)
>>> result = str(utool.dict_str(stat_dict))
>>> print(result)
{
    'max': 41.0,
    'min': 0.0,
    'mean': 20.0,
    'std': 13.177115,
    'nMin': 7,
    'nMax': 3,
    'shape': (100,),
    'num_nan': 1,
}
Examples1:
>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import numpy as np
>>> import utool
>>> axis = 0
>>> rng = np.random.RandomState(0)
>>> list_ = rng.randint(0, 42, size=100).astype(np.int32)
>>> stat_dict = get_stats(list_, axis, use_nan=True)
>>> result = str(utool.dict_str(stat_dict))
>>> print(result)
{
    'max': 41,
    'min': 0,
    'mean': 19.889999,
    'std': 13.156668,
    'nMin': 7,
    'nMax': 3,
    'shape': (100,),
    'num_nan': 0,
}
SeeAlso:
print_stats get_stats_str
utool.util_dev.get_stats_str(list_=None, newlines=False, keys=None, exclude_keys=[], lbl=None, precision=None, axis=0, stat_dict=None, use_nan=False, align=False, use_median=False, **kwargs)[source]

Returns the string version of get_stats

if keys is not None then it only displays chosen keys excluded keys are always removed

CommandLine:

python -m utool.util_dev --test-get_stats_str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> list_ = [1, 2, 3, 4, 5]
>>> newlines = False
>>> keys = None
>>> exclude_keys = []
>>> lbl = None
>>> precision = 2
>>> stat_str = get_stats_str(list_, newlines, keys, exclude_keys, lbl, precision)
>>> result = str(stat_str)
>>> print(result)
{'max': 5, 'min': 1, 'mean': 3, 'std': 1.41, 'nMin': 1, 'nMax': 1, 'shape': (5,)}
SeeAlso:
print_stats get_stats
utool.util_dev.get_submodules_from_dpath(dpath, only_packages=False, recursive=True)[source]
Parameters:
  • dpath (str) – directory path
  • only_packages (bool) – if True returns only package directories, otherwise returns module files. (default = False)
Returns:

submod_fpaths
Return type:

list

CommandLine:

python -m utool.util_dev --exec-get_submodules_from_dpath --only_packages

Example

>>> # SCRIPT
>>> from utool.util_dev import *  # NOQA
>>> import utool as ut
>>> dpath = ut.truepath_relative(ut.get_argval('--dpath', default='.'))
>>> print(dpath)
>>> only_packages = ut.get_argflag('--only_packages')
>>> submod_fpaths = get_submodules_from_dpath(dpath, only_packages)
>>> submod_fpaths = ut.lmap(ut.truepath_relative, submod_fpaths)
>>> result = ('submod_fpaths = %s' % (ut.repr3(submod_fpaths),))
>>> print(result)
utool.util_dev.grace_period(msg=u'', seconds=10)[source]

Gives user a window to stop a process before it happens

utool.util_dev.ifnone(default, value)[source]

shorthand for inline if / else statements

utool.util_dev.info(var, lbl)[source]
utool.util_dev.init_catch_ctrl_c()[source]
utool.util_dev.input_timeout(msg=u'Waiting for input...', timeout=30)[source]

FIXME: Function does not work quite right yet.

Parameters:
  • msg (str) –
  • timeout (int) –
Returns:

ans
Return type:

?

References

http://stackoverflow.com/questions/1335507/keyboard-input-with-timeout-in-python http://home.wlu.edu/~levys/software/kbhit.py http://stackoverflow.com/questions/3471461/raw-input-and-timeout/3911560#3911560

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> msg = 'Waiting for input...'
>>> timeout = 30
>>> ans = input_timeout(msg, timeout)
>>> print(ans)
utool.util_dev.inverable_group_multi_list(item_lists)[source]

aid_list1 = np.array([1, 1, 2, 2, 3, 3]) aid2_list = np.array([4, 2, 1, 9, 8, 7]) item_lists = (np.array(aid1_list), np.array(aid2_list))

utool.util_dev.inverable_unique_two_lists(item1_list, item2_list)[source]

item1_list = aid1_list item2_list = aid2_list

utool.util_dev.is_developer(mycomputers=None)[source]
utool.util_dev.iup()[source]

shortcut when pt is not imported

utool.util_dev.listinfo(list_, lbl=u'ndarr')[source]
utool.util_dev.make_at_least_n_items_valid(flag_list, n)[source]

tries to make at least min(len(flag_list, n) items True in flag_list

Parameters:
  • flag_list (list) – list of booleans
  • n (int) – number of items to ensure are True

CommandLine:

python -m utool.util_dev --test-make_at_least_n_items_valid

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> # build test data
>>> flag_list = [False, True, False, False, False, False, False, True]
>>> n = 5
>>> # execute function
>>> flag_list = make_at_least_n_items_valid(flag_list, n)
>>> # verify results
>>> result = str(flag_list)
>>> print(result)
[ True  True  True  True False False False  True]
utool.util_dev.make_call_graph(func, *args, **kwargs)[source]

profile with pycallgraph

Example

pycallgraph graphviz – ./mypythonscript.py

References

http://pycallgraph.slowchop.com/en/master/

utool.util_dev.make_object_graph(obj, fpath=u'sample_graph.png')[source]

memoryprofile with objgraph

Examples

#import objgraph #objgraph.show_most_common_types() #objgraph.show_growth() #memtrack.report() #memtrack.report() #objgraph.show_growth() #import gc #gc.collect() #memtrack.report() #y = 0 #objgraph.show_growth() #memtrack.report() #utool.embed()

References

http://mg.pov.lt/objgraph/

utool.util_dev.memory_dump()[source]

References

http://stackoverflow.com/questions/141351/how-do-i-find-what-is-using-memory-in-a-python-process-in-a-production-system

utool.util_dev.myprint(input_=None, prefix=u'', indent=u'', lbl=u'')[source]

OLD PRINT FUNCTION USED WITH PRINTABLE VAL TODO: Refactor and update

utool.util_dev.npArrInfo(arr)[source]

OLD update and refactor

utool.util_dev.npinfo(ndarr, lbl=u'ndarr')[source]
utool.util_dev.numpy_list_num_bits(nparr_list, expected_type, expected_dims)[source]
utool.util_dev.print_object_size(obj, lbl=u'')[source]
utool.util_dev.print_object_size_tree(obj, lbl=u'obj')[source]

Needs work

utool.util_dev.print_stats(list_, lbl=None, newlines=False, precision=2, axis=0, **kwargs)[source]

Prints string representation of stat of list_

CommandLine:

python -m utool.util_dev --test-print_stats

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> list_ = [1, 2, 3, 4, 5]
>>> lbl = None
>>> newlines = False
>>> precision = 2
>>> result = print_stats(list_, lbl, newlines, precision)
{'max': 5, 'min': 1, 'mean': 3, 'std': 1.41, 'nMin': 1, 'nMax': 1, 'shape': (5,)}
SeeAlso:
get_stats_str get_stats
utool.util_dev.printableType(val, name=None, parent=None)[source]

Tries to make a nice type string for a value. Can also pass in a Printable parent object

utool.util_dev.printableVal(val, type_bit=True, justlength=False)[source]

Very old way of doing pretty printing. Need to update and refactor. DEPRICATE

utool.util_dev.pylab_qt4()[source]
utool.util_dev.report_memsize(obj, name=None, verbose=True)[source]
utool.util_dev.reset_catch_ctrl_c()[source]
utool.util_dev.search_module(mod, pat, ignore_case=True, recursive=False, _seen=None)[source]

Searches module functions, classes, and constants for members matching a pattern.

Parameters:
  • mod (module) – live python module
  • pat (str) – regular expression
Returns:

found_list
Return type:

list

CommandLine:

python -m utool.util_dev --exec-search_module --mod=utool --pat=module
python -m utool.util_dev --exec-search_module --mod=opengm --pat=cut
python -m utool.util_dev --exec-search_module --mod=opengm --pat=multi
python -m utool.util_dev --exec-search_module --mod=plottool --pat=networkx
python -m utool.util_dev --exec-search_module --mod=utool --pat=Levenshtein

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import utool as ut
>>> recursive = True
>>> ignore_case = True
>>> modname = ut.get_argval('--mod', type_=str, default='utool')
>>> pat = ut.get_argval('--pat', type_=str, default='search')
>>> mod = ut.import_modname(modname)
>>> print('pat = %r' % (pat,))
>>> print('mod = %r' % (mod,))
>>> found_list = search_module(mod, pat, recursive=recursive)
>>> result = ('found_list = %s' % (ut.repr2(found_list),))
>>> print(result)
utool.util_dev.search_utool(pat)[source]
utool.util_dev.set_clipboard(text)[source]

References

http://stackoverflow.com/questions/11063458/python-script-to-copy-text-to-clipboard http://stackoverflow.com/questions/579687/how-do-i-copy-a-string-to-the-clipboard-on-windows-using-python

utool.util_dev.strip_line_comments(code_text, comment_char=u'#')[source]
utool.util_dev.testit(stmt, setup)[source]
utool.util_dev.timeit_compare(stmt_list, setup=u'', iterations=100000, verbose=True, strict=False)[source]

Compares several statments by timing them and also checks that they have the same return value

Parameters:
  • stmt_list (list) – list of statments to compare
  • setup (str) –
  • iterations (int) –
  • verbose (bool) –
  • strict (bool) –
Returns:

tuple – (passed, time_list, result_list) passed (bool): True if all results are the same time_list (list): list of times for each statment result_list (list): list of results values for each statment
Return type:

bool, list, list

CommandLine:

python -m utool.util_dev --exec-timeit_compare

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dev import *  # NOQA
>>> import utool as ut
>>> setup = ut.codeblock(
    '''
    import numpy as np
    rng = np.random.RandomState(0)
    invVR_mats = rng.rand(1000, 3, 3).astype(np.float64)
    ''')
>>> stmt1 = 'invVR_mats[:, 0:2, 2].T'
>>> stmt2 = 'invVR_mats.T[2, 0:2]'
>>> iterations = 1000
>>> verbose = True
>>> stmt_list = [stmt1, stmt2]
>>> ut.timeit_compare(stmt_list, setup=setup, iterations=iterations, verbose=verbose)
utool.util_dev.timeit_grid(stmt_list, setup=u'', iterations=10000, input_sizes=None, verbose=True, show=False)[source]
Timeit::

import utool as ut setup = ut.codeblock( ‘’’ import utool as ut from six.moves import range, zip import time def time_append(size):

start_time = time.time() last_time = start_time list2 = [] for x in range(size):

now_time = time.time() between = now_time - last_time last_time = now_time list2.append(between)
def time_assign(size):

start_time = time.time() last_time = start_time list1 = ut.alloc_nones(size) for x in range(size):

now_time = time.time() between = now_time - last_time last_time = now_time list1[x] = between
def time_baseline(size):

start_time = time.time() last_time = start_time for x in range(size):

now_time = time.time() between = now_time - last_time last_time = now_time
def time_null(size):
for x in range(size):
pass

‘’‘)

input_sizes = [2 ** count for count in range(7, 12)] stmt_list = [‘time_assign’, ‘time_append’, ‘time_baseline’, ‘time_null’] input_sizes=[100, 1000, 10000] ut.timeit_grid(stmt_list, setup, input_sizes=input_sizes, show=True)

utool.util_dev.tuples_to_unique_scalars(tup_list)[source]
utool.util_dev.uninvert_unique_two_lists(flat_list, reconstruct_tup)[source]

flat_list = thumb_list

utool.util_dev.user_cmdline_prompt(msg=u'')[source]

utool.util_dict module

convinience functions for dictionaries

class utool.util_dict.AutoVivification[source]

Bases: dict

Implementation of perl’s autovivification feature.

An AutoVivification is an infinitely nested default dict of dicts.

References

http://stackoverflow.com/questions/651794/best-way-to-init-dict-of-dicts

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> dict_ = AutoVivification()
>>> # Notice that there is no KeyError
>>> dict_[0][10][100] = None
>>> result = ('dict_ = %r' % (dict_,))
>>> print(result)
dict_ = {0: {10: {100: None}}}
class utool.util_dict.DictLike[source]

Bases: object

move to util_dict rectify with util_dev

Need to specify

getitem, setitem, keys

asdict()[source]
copy()[source]
items()[source]
iteritems()[source]
iterkeys()[source]
itervalues()[source]
values()[source]
utool.util_dict.all_dict_combinations(varied_dict)[source]
Parameters:varied_dict (dict) – a dict with lists of possible parameter settings
Returns:dict_list a list of dicts correpsonding to all combinations of params settings
Return type:list

CommandLine:

python -m utool.util_dict --test-all_dict_combinations

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> varied_dict = {'logdist_weight': [0.0, 1.0], 'pipeline_root': ['vsmany'], 'sv_on': [True, False, None]}
>>> dict_list = all_dict_combinations(varied_dict)
>>> result = str(ut.list_str(dict_list))
>>> print(result)
[
    {'logdist_weight': 0.0, 'pipeline_root': 'vsmany', 'sv_on': True},
    {'logdist_weight': 0.0, 'pipeline_root': 'vsmany', 'sv_on': False},
    {'logdist_weight': 0.0, 'pipeline_root': 'vsmany', 'sv_on': None},
    {'logdist_weight': 1.0, 'pipeline_root': 'vsmany', 'sv_on': True},
    {'logdist_weight': 1.0, 'pipeline_root': 'vsmany', 'sv_on': False},
    {'logdist_weight': 1.0, 'pipeline_root': 'vsmany', 'sv_on': None},
]
utool.util_dict.all_dict_combinations_lbls(varied_dict, remove_singles=True, allow_lone_singles=False)[source]

returns a label for each variation in a varydict.

It tries to not be oververbose and returns only what parameters are varied in each label.

CommandLine:

python -m utool.util_dict --test-all_dict_combinations_lbls
python -m utool.util_dict --exec-all_dict_combinations_lbls:1

Example

>>> # ENABLE_DOCTEST
>>> import utool
>>> from utool.util_dict import *  # NOQA
>>> varied_dict = {'logdist_weight': [0.0, 1.0], 'pipeline_root': ['vsmany'], 'sv_on': [True, False, None]}
>>> comb_lbls = utool.all_dict_combinations_lbls(varied_dict)
>>> result = (utool.list_str(comb_lbls))
>>> print(result)
[
    'logdist_weight=0.0,sv_on=True',
    'logdist_weight=0.0,sv_on=False',
    'logdist_weight=0.0,sv_on=None',
    'logdist_weight=1.0,sv_on=True',
    'logdist_weight=1.0,sv_on=False',
    'logdist_weight=1.0,sv_on=None',
]

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> from utool.util_dict import *  # NOQA
>>> varied_dict = {'logdist_weight': [0.0], 'pipeline_root': ['vsmany'], 'sv_on': [True]}
>>> allow_lone_singles = True
>>> comb_lbls = ut.all_dict_combinations_lbls(varied_dict, allow_lone_singles=allow_lone_singles)
>>> result = (ut.list_str(comb_lbls))
>>> print(result)
[
    'logdist_weight=0.0,pipeline_root=vsmany,sv_on=True',
]
utool.util_dict.all_dict_combinations_ordered(varied_dict)[source]

Same as all_dict_combinations but preserves order

utool.util_dict.assert_keys_are_subset(dict1, dict2)[source]

Example

>>> dict1 = {1:1, 2:2, 3:3}
>>> dict2 = {2:3, 3:3}
>>> assert_keys_are_subset(dict1, dict2)
>>> #dict2 = {4:3, 3:3}
utool.util_dict.augdict(dict1, dict2=None, **kwargs)[source]
utool.util_dict.build_conflict_dict(key_list, val_list)[source]

Builds dict where a list of values is associated with more than one key

Parameters:
  • key_list (list) –
  • val_list (list) –
Returns:

key_to_vals
Return type:

dict

CommandLine:

python -m utool.util_dict --test-build_conflict_dict

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> key_list = [  1,   2,   2,   3,   1]
>>> val_list = ['a', 'b', 'c', 'd', 'e']
>>> # execute function
>>> key_to_vals = build_conflict_dict(key_list, val_list)
>>> # verify results
>>> result = ut.dict_str(key_to_vals)
>>> print(result)
{
    1: ['a', 'e'],
    2: ['b', 'c'],
    3: ['d'],
}
utool.util_dict.count_dict_vals(dict_of_lists)[source]
utool.util_dict.delete_dict_keys(dict_, key_list)[source]

in place deletion if keys exist

Parameters:
  • dict –
  • key_list (list) –

CommandLine:

python -m utool.util_dict --test-delete_dict_keys

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> dict_ = {'bread': 1, 'churches': 1, 'cider': 2, 'very small rocks':2}
>>> key_list = ['duck', 'bread', 'cider']
>>> # execute function
>>> delete_dict_keys(dict_, key_list)
>>> # verify results
>>> result = ut.dict_str(dict_, nl=False)
>>> print(result)
{'churches': 1, 'very small rocks': 2}
utool.util_dict.delete_keys(dict_, key_list)

in place deletion if keys exist

Parameters:
  • dict –
  • key_list (list) –

CommandLine:

python -m utool.util_dict --test-delete_dict_keys

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> dict_ = {'bread': 1, 'churches': 1, 'cider': 2, 'very small rocks':2}
>>> key_list = ['duck', 'bread', 'cider']
>>> # execute function
>>> delete_dict_keys(dict_, key_list)
>>> # verify results
>>> result = ut.dict_str(dict_, nl=False)
>>> print(result)
{'churches': 1, 'very small rocks': 2}
utool.util_dict.dict_assign(dict_, keys, vals)[source]

simple method for assigning or setting values with a similar interface to dict_take

utool.util_dict.dict_filter_nones(dict_)[source]

Removes None values

Parameters:dict (dict) – a dictionary
Returns:
Return type:dict

CommandLine:

python -m utool.util_dict --exec-dict_filter_nones

Example

>>> # UNSTABLE_DOCTEST
>>> # fails on python 3 because of dict None order
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = {1: None, 2: 'blue', 3: 'four', None: 'fun'}
>>> dict2_ = dict_filter_nones(dict_)
>>> result = ut.dict_str(dict2_, nl=False)
>>> print(result)
{None: 'fun', 2: 'blue', 3: 'four'}
utool.util_dict.dict_find_keys(dict_, val_list)[source]
Parameters:
  • dict (dict) –
  • val_list (list) –
Returns:

found_dict
Return type:

dict

CommandLine:

python -m utool.util_dict --test-dict_find_keys

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> # build test data
>>> dict_ = {'default': 1, 'hierarchical': 5, 'linear': 0, 'kdtree': 1,
...          'composite': 3, 'autotuned': 255, 'saved': 254, 'kmeans': 2,
...          'lsh': 6, 'kdtree_single': 4}
>>> val_list = [1]
>>> # execute function
>>> found_dict = dict_find_keys(dict_, val_list)
>>> # verify results
>>> result = str(found_dict)
>>> print(result)
{1: ['kdtree', 'default']}
utool.util_dict.dict_find_other_sameval_keys(dict_, key)[source]

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> # build test data
>>> dict_ = {'default': 1, 'hierarchical': 5, 'linear': 0, 'kdtree': 1,
...          'composite': 3, 'autotuned': 255, 'saved': 254, 'kmeans': 2,
...          'lsh': 6, 'kdtree_single': 4}
>>> key = 'default'
>>> # execute function
>>> found_dict = dict_find_keys(dict_, val_list)
utool.util_dict.dict_hist(item_list, weight_list=None, ordered=False)[source]

Builds a histogram of items in item_list

Parameters:item_list (list) – list with hashable items (usually containing duplicates)
Returns:dictionary where the keys are items in item_list, and the values are the number of times the item appears in item_list.
Return type:dict

CommandLine:

python -m utool.util_dict --test-dict_hist

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> # build test data
>>> item_list = [1, 2, 39, 900, 1232, 900, 1232, 2, 2, 2, 900]
>>> # execute function
>>> hist_ = dict_hist(item_list)
>>> result = hist_
>>> # verify results
>>> print(result)
{1232: 2, 1: 1, 2: 4, 900: 3, 39: 1}
utool.util_dict.dict_hist_cumsum(hist_, reverse=True)[source]

VERY HACKY

utool.util_dict.dict_intersection(dict1, dict2, combine=False, combine_op=<built-in function add>)[source]
Parameters:
  • dict1 (dict) –
  • dict2 (dict) –
  • combine (bool) – Combines keys only if the values are equal if False else values are combined using combine_op (default = False)
  • combine_op (builtin_function_or_method) – (default = operator.add)
Returns:

mergedict_
Return type:

dict

CommandLine:

python -m utool.util_dict --exec-dict_intersection

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> dict2 = {'b': 2, 'c': 3, 'd': 5, 'e': 21, 'f': 42}
>>> combine = False
>>> mergedict_ = dict_intersection(dict1, dict2, combine)
>>> result = ('mergedict_ = %s' % (ut.dict_str(mergedict_, nl=False),))
>>> print(result)
mergedict_ = {'b': 2, 'c': 3}
utool.util_dict.dict_isect(dict1, dict2, combine=False, combine_op=<built-in function add>)
Parameters:
  • dict1 (dict) –
  • dict2 (dict) –
  • combine (bool) – Combines keys only if the values are equal if False else values are combined using combine_op (default = False)
  • combine_op (builtin_function_or_method) – (default = operator.add)
Returns:

mergedict_
Return type:

dict

CommandLine:

python -m utool.util_dict --exec-dict_intersection

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> dict2 = {'b': 2, 'c': 3, 'd': 5, 'e': 21, 'f': 42}
>>> combine = False
>>> mergedict_ = dict_intersection(dict1, dict2, combine)
>>> result = ('mergedict_ = %s' % (ut.dict_str(mergedict_, nl=False),))
>>> print(result)
mergedict_ = {'b': 2, 'c': 3}
utool.util_dict.dict_isect_combine(dict1, dict2, combine_op=<built-in function add>)[source]

Intersection of dict keys and combination of dict values

utool.util_dict.dict_keysubset(dict_, keys)[source]
utool.util_dict.dict_map_apply_vals(dict_, func)[source]

applies a function to each of the vals in dict_

Parameters:
  • dict (dict_) – a dictionary
  • func (function) –
Returns:
Return type:

dict_

CommandLine:

python -m utool.util_dict --test-dict_map_apply_vals

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> dict_ = {'a': [1, 2, 3], 'b': []}
>>> func = len
>>> result = dict_map_apply_vals(dict_, func)
>>> print(result)
{'a': 3, 'b': 0}
utool.util_dict.dict_setdiff(dict_, negative_keys)[source]

returns a copy of dict_ without keys in the negative_keys list

Parameters:
  • dict (dict) –
  • negative_keys (list) –
utool.util_dict.dict_stack(dict_list, key_prefix=u'')[source]

stacks values from two dicts into a new dict where the values are list of the input values. the keys are the same.

DEPRICATE in favor of dict_stack2

Parameters:dict_list (list) – list of dicts with similar keys
Returns:dict dict_stacked

CommandLine:

python -m utool.util_dict --test-dict_stack

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict1_ = {'a': 1, 'b': 2}
>>> dict2_ = {'a': 2, 'b': 3, 'c': 4}
>>> dict_stacked = dict_stack([dict1_, dict2_])
>>> result = ut.repr2(dict_stacked, sorted_=True)
>>> print(result)
{'a': [1, 2], 'b': [2, 3], 'c': [4]}
utool.util_dict.dict_stack2(dict_list, key_suffix=None, default=None)[source]

Stacks vals from a list of dicts into a dict of lists. Inserts Nones in place of empty items to preserve order.

Parameters:
  • dict_list (list) – list of dicts
  • key_suffix (str) – (default = None)
Returns:

stacked_dict
Return type:

dict
Setup:
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut

Example

>>> # ENABLE_DOCTEST
>>> # Usual case: multiple dicts as input
>>> dict1_ = {'a': 1, 'b': 2}
>>> dict2_ = {'a': 2, 'b': 3, 'c': 4}
>>> dict_list = [dict1_, dict2_]
>>> dict_stacked = dict_stack2(dict_list)
>>> result = ut.repr2(dict_stacked, sorted_=True)
>>> print(result)
{'a': [1, 2], 'b': [2, 3], 'c': [None, 4]}
Example1:
>>> # ENABLE_DOCTEST
>>> # Corner case: one dict as input
>>> dict1_ = {'a': 1, 'b': 2}
>>> dict_list = [dict1_]
>>> dict_stacked = dict_stack2(dict_list)
>>> result = ut.repr2(dict_stacked, sorted_=True)
>>> print(result)
{'a': [1], 'b': [2]}
Example2:
>>> # ENABLE_DOCTEST
>>> # Corner case: zero dicts as input
>>> dict_list = []
>>> dict_stacked = dict_stack2(dict_list)
>>> result = ut.repr2(dict_stacked, sorted_=True)
>>> print(result)
{}
Example3:
>>> # ENABLE_DOCTEST
>>> # Corner case: empty dicts as input
>>> dict_list = [{}]
>>> dict_stacked = dict_stack2(dict_list)
>>> result = ut.repr2(dict_stacked, sorted_=True)
>>> print(result)
{}
Example3:
>>> # ENABLE_DOCTEST
>>> # Corner case: 3 dicts
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_list = [{'a': 1}, {'b': 1}, {'c': 1}, {'b': 2}]
>>> default = None
>>> dict_stacked = dict_stack2(dict_list, default=default)
>>> result = ut.repr2(dict_stacked, sorted_=True)
>>> print(result)
{'a': [1, None, None, None], 'b': [None, 1, None, 2], 'c': [None, None, 1, None]}
utool.util_dict.dict_subset(dict_, keys, *d)[source]
Parameters:
  • dict (dict) –
  • keys (list) –
Returns:

subset dictionary
Return type:

dict

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = {'K': 3, 'dcvs_clip_max': 0.2, 'p': 0.1}
>>> keys = ['K', 'dcvs_clip_max']
>>> d = tuple([])
>>> subdict_ = dict_subset(dict_, keys)
>>> result = ut.dict_str(subdict_, sorted_=True, newlines=False)
>>> print(result)
{'K': 3, 'dcvs_clip_max': 0.2}
utool.util_dict.dict_take(dict_, keys, *d)[source]

get multiple values from a dictionary

utool.util_dict.dict_take_asnametup(dict_, keys, name=u'_NamedTup')[source]
utool.util_dict.dict_take_gen(dict_, keys, *d)[source]

generate multiple values from a dictionary

Parameters:
  • dict (dict) –
  • keys (list) –
Varargs:
d: if specified is default for key errors

CommandLine:

python -m utool.util_dict --test-dict_take_gen
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = {1: 'a', 2: 'b', 3: 'c'}
>>> keys = [1, 2, 3, 4, 5]
>>> result = list(dict_take_gen(dict_, keys, None))
>>> result = ut.list_str(result, nl=False)
>>> print(result)
['a', 'b', 'c', None, None]
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> dict_ = {1: 'a', 2: 'b', 3: 'c'}
>>> keys = [1, 2, 3, 4, 5]
>>> try:
>>>     print(list(dict_take_gen(dict_, keys)))
>>>     result = 'did not get key error'
>>> except KeyError:
>>>     result = 'correctly got key error'
>>> print(result)
correctly got key error
utool.util_dict.dict_take_list(dict_, keys, *d)

get multiple values from a dictionary

utool.util_dict.dict_take_pop(dict_, keys, *d)[source]

like dict_take but pops values off

CommandLine:

python -m utool.util_dict --test-dict_take_pop
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = {1: 'a', 'other': None, 'another': 'foo', 2: 'b', 3: 'c'}
>>> keys = [1, 2, 3, 4, 5]
>>> print('before: ' + ut.dict_str(dict_))
>>> result = list(dict_take_pop(dict_, keys, None))
>>> result = ut.list_str(result, nl=False)
>>> print('after: ' + ut.dict_str(dict_))
>>> assert len(dict_) == 2
>>> print(result)
['a', 'b', 'c', None, None]
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = {1: 'a', 2: 'b', 3: 'c'}
>>> keys = [1, 2, 3, 4, 5]
>>> print('before: ' + ut.dict_str(dict_))
>>> try:
>>>     print(list(dict_take_pop(dict_, keys)))
>>>     result = 'did not get key error'
>>> except KeyError:
>>>     result = 'correctly got key error'
>>> assert len(dict_) == 0
>>> print('after: ' + ut.dict_str(dict_))
>>> print(result)
correctly got key error
utool.util_dict.dict_to_keyvals(dict_)[source]
utool.util_dict.dict_union(*args)[source]
utool.util_dict.dict_union2(dict1, dict2)[source]
utool.util_dict.dict_union3(dict1, dict2, combine_op=<built-in function add>)[source]
Parameters:
  • dict1 (dict) –
  • dict2 (dict) –
  • combine_op (builtin_function_or_method) – (default = operator.add)
Returns:

mergedict_
Return type:

dict

CommandLine:

python -m utool.util_dict --exec-dict_union3

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict1 = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> dict2 = {'b': 2, 'c': 3, 'd': 5, 'e': 21, 'f': 42}
>>> combine_op = operator.add
>>> mergedict_ = dict_union3(dict1, dict2, combine_op)
>>> result = ('mergedict_ = %s' % (ut.dict_str(mergedict_, nl=False),))
>>> print(result)
mergedict_ = {'a': 1, 'b': 4, 'c': 6, 'd': 9, 'e': 21, 'f': 42}
utool.util_dict.dict_union_combine(dict1, dict2, combine_op=<built-in function add>, default=<utool.util_const.ClassNoParam object>, default2=<utool.util_const.ClassNoParam object>)[source]

Combine of dict keys and uses dfault value when key does not exist

CAREFUL WHEN USING THIS WITH REDUCE. Use dict_stack2 instead

utool.util_dict.dict_update_newkeys(dict_, dict2)[source]

Like dict.update, but does not overwrite items

utool.util_dict.dict_val_map(dict_, func)

applies a function to each of the vals in dict_

Parameters:
  • dict (dict_) – a dictionary
  • func (function) –
Returns:
Return type:

dict_

CommandLine:

python -m utool.util_dict --test-dict_map_apply_vals

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> dict_ = {'a': [1, 2, 3], 'b': []}
>>> func = len
>>> result = dict_map_apply_vals(dict_, func)
>>> print(result)
{'a': 3, 'b': 0}
utool.util_dict.dict_where_len0(dict_)[source]

Accepts a dict of lists. Returns keys that have vals with no length

utool.util_dict.dictinfo(dict_)[source]

In depth debugging info

Parameters:dict (dict) –
Returns:str

Example

>>> from utool.util_dict import *  # NOQA
>>> dict_ = {}
>>> result = dictinfo(dict_)
>>> print(result)
utool.util_dict.get_dict_column(dict_, colx)[source]
Parameters:
  • dict (dict_) – a dictionary
  • colx –

CommandLine:

python -m utool.util_dict --test-get_dict_column

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> dict_ = {'a': [0, 1, 2], 'b': [3, 4, 5], 'c': [6, 7, 8]}
>>> colx = [2, 0]
>>> retdict_ = get_dict_column(dict_, colx)
>>> result = str(retdict_)
>>> print(result)
{'a': [2, 0], 'c': [8, 6], 'b': [5, 3]}
utool.util_dict.get_dict_hashid(dict_)[source]
Parameters:dict (dict) –
Returns:id hash
Return type:int

References

http://stackoverflow.com/questions/5884066/hashing-a-python-dictionary

CommandLine:

python -m utool.util_dict --test-get_dict_hashid
python3 -m utool.util_dict --test-get_dict_hashid

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> dict_ = {}
>>> dict_ = {'a': 'b'}
>>> dict_ = {'a': {'c': 'd'}}
>>> #dict_ = {'a': {'c': 'd'}, 1: 143, dict: set}
>>> dict_ = {'a': {'c': 'd'}, 1: 143 }
>>> hashid = get_dict_hashid(dict_)
>>> result = str(hashid)
>>> print(result)
oegknoalkrkojumi

fprzgbpsdzfqueqh

5127623379007436803

utool.util_dict.group_items(item_list, groupid_list, sorted_=True)[source]
Parameters:
  • item_list (list) –
  • groupid_list (list) –
  • sorted (bool) – if True preserves the ordering of items within groups (default = True)
Returns:

groupid2_items mapping groupids to a list of items
Return type:

dict
SeeAlso:
vtool.group_indices - much faster numpy grouping algorithm vtool.apply_gropuing - second part to faster numpy grouping algorithm

CommandLine:

python -m utool.util_dict --exec-group_items

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> item_list    = [ 'ham',      'jam',    'spam',     'eggs', 'cheese', 'bannana']
>>> groupid_list = ['protein', 'fruit', 'protein',  'protein',  'dairy',   'fruit']
>>> groupid2_items = ut.group_items(item_list, iter(groupid_list))
>>> result = ut.dict_str(groupid2_items, nl=False, strvals=False)
>>> print(result)
{'dairy': ['cheese'], 'fruit': ['jam', 'bannana'], 'protein': ['ham', 'spam', 'eggs']}
utool.util_dict.groupby_attr(item_list, attrname)[source]
utool.util_dict.groupby_tags(item_list, tags_list)[source]

case where an item can belong to multiple groups

Parameters:
  • item_list (list) –
  • tags_list (list) –
Returns:

groupid2_items
Return type:

dict

CommandLine:

python -m utool.util_dict --exec-groupby_tags

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> tagged_item_list = {
>>>     'spam': ['meat', 'protein', 'food'],
>>>     'eggs': ['protein', 'food'],
>>>     'cheese': ['dairy', 'protein', 'food'],
>>>     'jam':  ['fruit', 'food'],
>>>     'banana': ['weapon', 'fruit', 'food'],
>>> }
>>> item_list = list(tagged_item_list.keys())
>>> tags_list = list(tagged_item_list.values())
>>> groupid2_items = groupby_tags(item_list, tags_list)
>>> result = ('groupid2_items = %s' % (ut.dict_str(groupid2_items),))
>>> print(result)
groupid2_items = {
    'dairy': ['cheese'],
    'food': ['cheese', 'eggs', 'jam', 'banana', 'spam'],
    'fruit': ['jam', 'banana'],
    'meat': ['spam'],
    'protein': ['cheese', 'eggs', 'spam'],
    'weapon': ['banana'],
}
utool.util_dict.hierarchical_group_items(item_list, groupids_list)[source]

Generalization of group_item. Convert a flast list of ids into a heirarchical dictionary.

TODO: move to util_dict

Reference:
http://stackoverflow.com/questions/10193235/python-translate-a-table-to-a-hierarchical-dictionary
Parameters:
  • item_list (list) –
  • groupids_list (list) –

CommandLine:

python -m utool.util_dict --exec-hierarchical_group_items

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> item_list     = [1, 2, 3, 4]
>>> groupids_list = [[1, 1, 2, 2]]
>>> tree = hierarchical_group_items(item_list, groupids_list)
>>> result = ('tree = ' + ut.dict_str(tree, nl=len(groupids_list) - 1))
>>> print(result)
tree = {1: [1, 2], 2: [3, 4]}

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> item_list     = [1, 2, 3, 4, 5, 6, 7, 8]
>>> groupids_list = [[1, 2, 1, 2, 1, 2, 1, 2], [3, 2, 2, 2, 3, 1, 1, 1]]
>>> tree = hierarchical_group_items(item_list, groupids_list)
>>> result = ('tree = ' + ut.dict_str(tree, nl=len(groupids_list) - 1))
>>> print(result)
tree = {
    1: {1: [7], 2: [3], 3: [1, 5]},
    2: {1: [6, 8], 2: [2, 4]},
}

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> item_list     = [1, 2, 3, 4]
>>> groupids_list = [[1, 1, 1, 2], [1, 2, 2, 2], [1, 3, 1, 1]]
>>> tree = hierarchical_group_items(item_list, groupids_list)
>>> result = ('tree = ' + ut.dict_str(tree, nl=len(groupids_list) - 1))
>>> print(result)
tree = {
    1: {
        1: {1: [1]},
        2: {1: [3], 3: [2]},
    },
    2: {
        2: {1: [4]},
    },
}
utool.util_dict.hierarchical_map_vals(func, node, max_depth=None, depth=0)[source]

node is a dict tree like structure with leaves of type list

TODO: move to util_dict

CommandLine:

python -m utool.util_dict --exec-hierarchical_map_vals

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> item_list     = [1, 2, 3, 4, 5, 6, 7, 8]
>>> groupids_list = [[1, 2, 1, 2, 1, 2, 1, 2], [3, 2, 2, 2, 3, 1, 1, 1]]
>>> tree = ut.hierarchical_group_items(item_list, groupids_list)
>>> len_tree = ut.hierarchical_map_vals(len, tree)
>>> result = ('len_tree = ' + ut.dict_str(len_tree, nl=1))
>>> print(result)
len_tree = {
    1: {1: 1, 2: 1, 3: 2},
    2: {1: 2, 2: 2},
}
Example1:
>>> # UNSTABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> depth = 4
>>> item_list = list(range(2 ** (depth + 1)))
>>> num = len(item_list) // 2
>>> groupids_list = []
>>> total = 0
>>> for level in range(depth):
...     num2 = len(item_list) // int((num * 2))
...     #nonflat_levelids = [([total + 2 * x + 1] * num + [total + 2 * x + 2] * num) for x in range(num2)]
...     nonflat_levelids = [([1] * num + [2] * num) for x in range(num2)]
...     levelids = ut.flatten(nonflat_levelids)
...     groupids_list.append(levelids)
...     total += num2 * 2
...     num //= 2
>>> print('groupids_list = %s' % (ut.list_str(groupids_list, nl=1),))
>>> print('depth = %r' % (len(groupids_list),))
>>> tree = ut.hierarchical_group_items(item_list, groupids_list)
>>> print('tree = ' + ut.dict_str(tree, nl=None))
>>> flat_tree_values = list(ut.iflatten_dict_values(tree))
>>> assert sorted(flat_tree_values) == sorted(item_list)
>>> print('flat_tree_values = ' + str(flat_tree_values))
>>> #print('flat_tree_keys = ' + str(list(ut.iflatten_dict_keys(tree))))
>>> #print('iflatten_dict_items = ' + str(list(ut.iflatten_dict_items(tree))))
>>> len_tree = ut.hierarchical_map_vals(len, tree, max_depth=4)
>>> result = ('len_tree = ' + ut.dict_str(len_tree, nl=None))
>>> print(result)
utool.util_dict.hmap_vals(func, node, max_depth=None, depth=0)

node is a dict tree like structure with leaves of type list

TODO: move to util_dict

CommandLine:

python -m utool.util_dict --exec-hierarchical_map_vals

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> item_list     = [1, 2, 3, 4, 5, 6, 7, 8]
>>> groupids_list = [[1, 2, 1, 2, 1, 2, 1, 2], [3, 2, 2, 2, 3, 1, 1, 1]]
>>> tree = ut.hierarchical_group_items(item_list, groupids_list)
>>> len_tree = ut.hierarchical_map_vals(len, tree)
>>> result = ('len_tree = ' + ut.dict_str(len_tree, nl=1))
>>> print(result)
len_tree = {
    1: {1: 1, 2: 1, 3: 2},
    2: {1: 2, 2: 2},
}
Example1:
>>> # UNSTABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> depth = 4
>>> item_list = list(range(2 ** (depth + 1)))
>>> num = len(item_list) // 2
>>> groupids_list = []
>>> total = 0
>>> for level in range(depth):
...     num2 = len(item_list) // int((num * 2))
...     #nonflat_levelids = [([total + 2 * x + 1] * num + [total + 2 * x + 2] * num) for x in range(num2)]
...     nonflat_levelids = [([1] * num + [2] * num) for x in range(num2)]
...     levelids = ut.flatten(nonflat_levelids)
...     groupids_list.append(levelids)
...     total += num2 * 2
...     num //= 2
>>> print('groupids_list = %s' % (ut.list_str(groupids_list, nl=1),))
>>> print('depth = %r' % (len(groupids_list),))
>>> tree = ut.hierarchical_group_items(item_list, groupids_list)
>>> print('tree = ' + ut.dict_str(tree, nl=None))
>>> flat_tree_values = list(ut.iflatten_dict_values(tree))
>>> assert sorted(flat_tree_values) == sorted(item_list)
>>> print('flat_tree_values = ' + str(flat_tree_values))
>>> #print('flat_tree_keys = ' + str(list(ut.iflatten_dict_keys(tree))))
>>> #print('iflatten_dict_items = ' + str(list(ut.iflatten_dict_items(tree))))
>>> len_tree = ut.hierarchical_map_vals(len, tree, max_depth=4)
>>> result = ('len_tree = ' + ut.dict_str(len_tree, nl=None))
>>> print(result)
utool.util_dict.iflatten_dict_values(node, depth=0)[source]
utool.util_dict.invert_dict(dict_)[source]
Parameters:dict (dict_) –
Returns:inverted_dict
Return type:dict

CommandLine:

python -m utool.util_dict --test-invert_dict

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = {'a': 1, 'b': 2}
>>> inverted_dict = invert_dict(dict_)
>>> result = ut.dict_str(inverted_dict, nl=False)
>>> print(result)
{1: 'a', 2: 'b'}

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = OrderedDict([(2, 'good',), (1, 'ok',), (0, 'junk',), (None, 'UNKNOWN',),])
>>> inverted_dict = invert_dict(dict_)
>>> result = ut.dict_str(inverted_dict, nl=False)
>>> print(result)
{'good': 2, 'ok': 1, 'junk': 0, 'UNKNOWN': None}
utool.util_dict.is_dicteq(dict1_, dict2_, almosteq_ok=True, verbose_err=True)[source]

Checks to see if dicts are the same. Performs recursion. Handles numpy

utool.util_dict.items_sorted_by_value(dict_)[source]
utool.util_dict.iter_all_dict_combinations_ordered(varied_dict)[source]

Same as all_dict_combinations but preserves order

utool.util_dict.iteritems_sorted(dict_)[source]

change to iteritems ordered

utool.util_dict.keys_sorted_by_value(dict_)[source]
utool.util_dict.map_dict_keys(func, dict_)[source]

a better version of dict_map_apply_vals

utool.util_dict.map_dict_vals(func, dict_)[source]

probably a better version of dict_map_apply_vals

utool.util_dict.merge_dicts(*args)[source]

add / concatenate / union / join / merge / combine dictionaries

Copies the first dictionary given and then repeatedly calls update using the rest of the dicts given in args. Duplicate keys will receive the last value specified the list of dictionaries.

Returns:mergedict_
Return type:dict

CommandLine:

python -m utool.util_dict --test-merge_dicts

References

http://stackoverflow.com/questions/38987/how-can-i-merge-two-python-dictionaries-in-a-single-expression

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> x = {'a': 1, 'b': 2}
>>> y = {'b': 3, 'c': 4}
>>> mergedict_ = merge_dicts(x, y)
>>> result = ut.dict_str(mergedict_, sorted_=True, newlines=False)
>>> print(result)
{'a': 1, 'b': 3, 'c': 4}
utool.util_dict.move_odict_item(odict, key, newpos)[source]

References

http://stackoverflow.com/questions/22663966/changing-order-of-ordered-dictionary-in-python

CommandLine:

python -m utool.util_dict --exec-move_odict_item

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> odict = OrderedDict()
>>> odict['a'] = 1
>>> odict['b'] = 2
>>> odict['c'] = 3
>>> odict['e'] = 5
>>> print(ut.dict_str(odict, nl=False))
>>> move_odict_item(odict, 'c', 1)
>>> print(ut.dict_str(odict, nl=False))
>>> move_odict_item(odict, 'a', 3)
>>> print(ut.dict_str(odict, nl=False))
>>> move_odict_item(odict, 'a', 0)
>>> print(ut.dict_str(odict, nl=False))
>>> move_odict_item(odict, 'b', 2)
>>> result = ut.dict_str(odict, nl=False)
>>> print(result)
{'a': 1, 'c': 3, 'b': 2, 'e': 5}
utool.util_dict.order_dict_by(dict_, key_order)[source]
Parameters:
  • dict (dict_) – a dictionary
  • key_order (list) –
Returns:

sorted_dict
Return type:

dict

CommandLine:

python -m utool.util_dict --exec-order_dict_by

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> import utool as ut
>>> dict_ = {1:1, 2:2, 3:3, 4:4}
>>> key_order = [4,2,3,1]
>>> sorted_dict = order_dict_by(dict_, key_order)
>>> result = ('sorted_dict = %s' % (ut.dict_str(sorted_dict, nl=False),))
>>> print(result)
sorted_dict = {4: 4, 2: 2, 3: 3, 1: 1}
utool.util_dict.sort_dict(dict_)[source]

enforces ordering on dict

utool.util_dict.update_existing(dict1, dict2, copy=False, assert_exists=False, iswarning=False, alias_dict=None)[source]

updates vals in dict1 using vals from dict2 only if the key is already in dict1.

Parameters:
  • dict1 (dict) –
  • dict2 (dict) –
  • copy (bool) – if true modifies dictionary in place (default = False)
  • assert_exists (bool) – if True throws error if new key specified (default = False)
  • dict1 –
  • dict2 –
  • copy – (default = False)
  • assert_exists – (default = False)
  • alias_dict (dict) – dictionary of alias keys for dict2 (default = None)

CommandLine:

python -m utool.util_dict --test-update_existing

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_dict import *  # NOQA
>>> # build test data
>>> dict1 = {'a': 1, 'b': 2, 'c': 3}
>>> dict2 = {'a': 2, 'd': 3}
>>> # execute function
>>> dict1_ = update_existing(dict1, dict2)
>>> assert 'd' not in dict1
>>> assert dict1['a'] == 2
>>> assert dict1_ is dict1
utool.util_dict.updateif_haskey(*args, **kwargs)[source]

utool.util_distances module

# TODO DEPRICATE AND MOVE TO VTOOL

utool.util_distances.nearest_point(x, y, pts, mode='random')[source]

finds the nearest point(s) in pts to (x, y) FIXME VIZ FEATROW

utool.util_func module

utool.util_func.compose_functions(*func_list)[source]
Referenes:
https://mathieularose.com/function-composition-in-python/
utool.util_func.general_get(getter, index, **kwargs)[source]

Works with getter funcs or indexable read/write arrays

utool.util_func.general_set(setter, index, val, **kwargs)[source]

Works with setter funcs or indexable read/write arrays

utool.util_func.identity(input_)[source]

identity function

utool.util_func.uinput_1to1(func, input_)[source]

universal input (really just accept list or tuple as input to a list only function)

Move to guitool

utool.util_git module

TODO: export from utool

utool.util_git.checkout_repos(repo_urls, repo_dirs=None, checkout_dir=None)[source]

Checkout every repo in repo_urls into checkout_dir

utool.util_git.ensure_project_repos()[source]
utool.util_git.ensure_repos(repo_urls, repo_dirs=None, checkout_dir=None)[source]

Checkout every repo in repo_urls into checkout_dir

utool.util_git.get_project_repo_dirs()[source]
utool.util_git.gg_command(command, sudo=False, repo_dirs=None)[source]

Runs a command on all of your PROJECT_REPO_DIRS

utool.util_git.git_sequence_editor_squash(fpath)[source]

squashes wip messages

CommandLine:

python -m utool.util_git --exec-git_sequence_editor_squash

Example

>>> # SCRIPT
>>> import utool as ut
>>> from utool.util_git import *  # NOQA
>>> fpath = ut.get_argval('--fpath', str, default=None)
>>> git_sequence_editor_squash(fpath)
utool.util_git.is_gitrepo(repo_dir)[source]
utool.util_git.pull_repos(repo_dirs, repos_with_submodules=[])[source]
utool.util_git.rename_branch(old_branch_name, new_branch_name, repo='.', remote='origin', dryrun=False)[source]

References

http://stackoverflow.com/questions/1526794/rename-master-branch-for-both-local-and-remote-git-repositories?answertab=votes#tab-top http://stackoverflow.com/questions/9524933/renaming-a-branch-in-github

CommandLine:

python -m utool.util_git --test-rename_branch --old=mymaster --new=ibeis_master --dryrun

Example

>>> # SCRIPT
>>> from utool.util_git import *  # NOQA
>>> repo = ut.get_argval('--repo', str, '.')
>>> remote = ut.get_argval('--remote', str, 'origin')
>>> old_branch_name = ut.get_argval('--old', str, None)
>>> new_branch_name = ut.get_argval('--new', str, None)
>>> rename_branch(old_branch_name, new_branch_name, repo, remote)
utool.util_git.repocmd(repo, command, sudo=False, dryrun=False)[source]

TODO change name to repo command runs a command on a repo

utool.util_git.set_code_dir(code_dir)[source]
utool.util_git.set_project_repos(repo_urls, repo_dirs)[source]
utool.util_git.setup_develop_repos(repo_dirs)[source]

Run python installs

utool.util_git.std_build_command(repo='.')[source]

My standard build script names.

Calls mingw_build.bat on windows and unix_build.sh on unix

utool.util_grabdata module

utool.util_grabdata.archive_files(archive_fpath, fpath_list, small=True, allowZip64=False, overwrite=False, verbose=True, common_prefix=False)[source]

Adds the files in fpath_list to an zip/tar archive.

Parameters:
  • archive_fpath (str) – path to zipfile to create
  • fpath_list (list) – path of files to add to the zipfile
  • small (bool) – if True uses compression but the zipfile will take more time to write
  • allowZip64 (bool) – use if a file is over 2GB
  • overwrite (bool) –
  • verbose (bool) – verbosity flag(default = True)
  • common_prefix (bool) – (default = False)

References

https://docs.python.org/2/library/zipfile.html

CommandLine:

python -m utool.util_grabdata --test-archive_files

Example

>>> # SLOW_DOCTEST
>>> from utool.util_grabdata import *  # NOQA
>>> import utool as ut
>>> archive_fpath = ut.get_app_resource_dir('utool', 'testarchive.zip')
>>> # remove an existing test archive
>>> ut.delete(archive_fpath)
>>> assert not exists(archive_fpath), 'archive should not exist'
>>> fpath_list = [ut.grab_test_imgpath(key) for key in ut.TESTIMG_URL_DICT]
>>> small = True
>>> allowZip64 = False
>>> overwrite = True
>>> result = archive_files(archive_fpath, fpath_list, small, allowZip64, overwrite)
>>> # verify results
>>> print(result)
>>> assert exists(archive_fpath), 'archive should exist'

Dropbox links should be en-mass downloaed from dl.dropbox

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_grabdata import *  # NOQA
>>> dropbox_url = 'www.dropbox.com/foobar.zip?dl=0'
>>> cleaned_url = clean_dropbox_link(dropbox_url)
>>> result = str(cleaned_url)
>>> print(result)
dl.dropbox.com/foobar.zip
utool.util_grabdata.clear_test_img_cache()[source]

CommandLine:

python -m utool.util_grabdata --test-clear_test_img_cache

Example

>>> # UNSTABLE_DOCTEST
>>> from utool.util_grabdata import *  # NOQA
>>> testimg_fpath = clear_test_img_cache()
>>> result = str(testimg_fpath)
>>> print(result)
utool.util_grabdata.download_url(url, filename=None, spoof=False)[source]

downloads a url to a filename.

Parameters:
  • url (str) – url to download
  • filename (str) – path to download to. Defaults to basename of url
  • spoof (bool) – if True pretends to by Firefox

References

http://blog.moleculea.com/2012/10/04/urlretrieve-progres-indicator/

TODO:
Delete any partially downloaded files

Example

>>> from utool.util_grabdata import *  # NOQA
>>> url = 'http://www.jrsoftware.org/download.php/ispack.exe'
>>> fpath = download_url(url)
>>> print(fpath)
ispack.exe
utool.util_grabdata.experiment_download_multiple_urls(url_list)[source]

References

http://stackoverflow.com/questions/1112343/capture-sigint-in-python http://stackoverflow.com/questions/16694907/download-large-file-requests GracefulInterruptHandler

Example

>>> # UNSTABLE_DOCTEST
>>> url_list = [
>>>     'https://www.dropbox.com/s/jl506apezj42zjz/ibeis-win32-setup-ymd_hm-2015-08-01_16-28.exe',   # NOQA
>>>     'https://www.dropbox.com/s/v1ivnmny6tlc364/vgg.caffe.slice_0_30_None.pickle',
>>>     'https://www.dropbox.com/s/v1ivnmny6tlc364/vgg.caffe.slice_0_30_None.pickle',
>>>     'https://www.dropbox.com/s/v1ivnmny6tlc364/vgg.caffe.slice_0_30_None.pickle',
>>>     'https://www.dropbox.com/s/v1ivnmny6tlc364/vgg.caffe.slice_0_30_None.pickle',
>>>     'https://www.dropbox.com/s/v1ivnmny6tlc364/vgg.caffe.slice_0_30_None.pickle',
>>>     ]
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/L10/L10_R1/S1_L10_R1_PICT0070.JPG'
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0001.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0002.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0003.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0004.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0005.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0006.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0007.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0008.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0022.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0023.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0024.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0025.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0026.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0027.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0028.JPG',
>>>     'https://snapshotserengeti.s3.msi.umn.edu/S1/B04/B04_R1/S1_B04_R1_PICT0029.JPG'
>>> ]
utool.util_grabdata.geo_locate(default='Unknown', timeout=1)[source]
utool.util_grabdata.get_prefered_browser(pref_list=[], fallback=True)[source]
Parameters:
  • browser_preferences (list) – (default = [])
  • fallback (bool) – uses default if non of preferences work (default = True)

CommandLine:

python -m utool.util_grabdata --test-get_prefered_browser

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_grabdata import *  # NOQA
>>> browser_preferences = ['firefox', 'chrome', 'safari']
>>> fallback = True
>>> browser = get_prefered_browser(browser_preferences, fallback)
>>> result = ('browser = %s' % (str(browser),))
>>> print(result)
>>> ut.quit_if_noshow()
utool.util_grabdata.get_valid_test_imgkeys()[source]

returns valid keys for grab_test_imgpath

utool.util_grabdata.grab_file_url(file_url, ensure=True, appname='utool', download_dir=None, delay=None, spoof=False, fname=None, verbose=True, redownload=False)[source]

Downloads a file and returns the local path of the file.

The resulting file is cached, so multiple calls to this function do not result in multiple dowloads.

Parameters:
  • file_url (str) – url to the file
  • ensure (bool) – if False the file is assumed to be downloaed (default = True)
  • appname (str) – (default = ‘utool’)
  • custom directory (download_dir) – (default = None)
  • delay (None) – delay time before download (default = None)
  • spoof (bool) – (default = False)
  • fname (str) – custom file name (default = None)
  • verbose (bool) – verbosity flag (default = True)
  • redownload (bool) – if True forces redownload of the file (default = False)
Returns:

fpath
Return type:

str

CommandLine:

sh -c "python ~/code/utool/utool/util_grabdata.py --all-examples"
python -m utool.util_grabdata --test-grab_file_url

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_grabdata import *  # NOQA
>>> import utool as ut  # NOQA
>>> from os.path import basename
>>> file_url = 'http://i.imgur.com/JGrqMnV.png'
>>> ensure = True
>>> appname = 'utool'
>>> download_dir = None
>>> delay = None
>>> spoof = False
>>> verbose = True
>>> redownload = True
>>> fname ='lena.png'
>>> lena_fpath = ut.grab_file_url(file_url, ensure, appname, download_dir,
>>>     delay, spoof, fname, verbose, redownload)
>>> result = basename(lena_fpath)
>>> print(result)
lena.png
utool.util_grabdata.grab_s3_contents(fpath, bucket, key, auth_access_id=None, auth_secret_key=None, auth_domain=None)[source]
utool.util_grabdata.grab_selenium_chromedriver()[source]

Automatically download selenium chrome driver if needed

CommandLine:

python -m utool.util_grabdata --test-grab_selenium_chromedriver:1

Example

>>> # DISABLE_DOCTEST
>>> ut.grab_selenium_chromedriver()
>>> import selenium.webdriver
>>> driver = selenium.webdriver.Chrome()
>>> driver.get('http://www.google.com')
>>> search_field = driver.find_element_by_name('q')
>>> search_field.send_keys('puppies')
>>> search_field.send_keys(selenium.webdriver.common.keys.Keys.ENTER)
Example1:
>>> # DISABLE_DOCTEST
>>> import selenium.webdriver
>>> driver = selenium.webdriver.Firefox()
>>> driver.get('http://www.google.com')
>>> search_field = driver.find_element_by_name('q')
>>> search_field.send_keys('puppies')
>>> search_field.send_keys(selenium.webdriver.common.keys.Keys.ENTER)
utool.util_grabdata.grab_selenium_driver(driver_name=None)[source]
utool.util_grabdata.grab_test_imgpath(key='lena.png', allow_external=True, verbose=True)[source]

Gets paths to standard / fun test images. Downloads them if they dont exits

Parameters:
  • key (str) – one of the standard test images, e.g. lena.png, carl.jpg, ...
  • allow_external (bool) – if True you can specify existing fpaths
Returns:

testimg_fpath - filepath to the downloaded or cached test image.
Return type:

str
SeeAlso:
ut.get_valid_test_imgkeys

CommandLine:

python -m utool.util_grabdata --test-grab_test_imgpath

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_grabdata import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> key = 'carl.jpg'
>>> # execute function
>>> testimg_fpath = grab_test_imgpath(key)
>>> # verify results
>>> ut.assertpath(testimg_fpath)
utool.util_grabdata.grab_zipped_url(zipped_url, ensure=True, appname='utool', download_dir=None, force_commonprefix=True, cleanup=False, redownload=False, spoof=False)[source]

downloads and unzips the url

Parameters:
  • zipped_url (str) – url which must be either a .zip of a .tar.gz file
  • ensure (bool) – eager evaluation if True(default = True)
  • appname (str) – (default = ‘utool’)
  • download_dir (str) – containing downloading directory
  • force_commonprefix (bool) – (default = True)
  • cleanup (bool) – (default = False)
  • redownload (bool) – (default = False)
  • spoof (bool) – (default = False)

CommandLine:

python -m utool.util_grabdata --exec-grab_zipped_url --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_grabdata import *  # NOQA
>>> import utool as ut
>>> zipped_url = '?'
>>> ensure = True
>>> appname = 'utool'
>>> download_dir = None
>>> force_commonprefix = True
>>> cleanup = False
>>> redownload = False
>>> spoof = False
>>> result = grab_zipped_url(zipped_url, ensure, appname, download_dir,
>>>                          force_commonprefix, cleanup, redownload,
>>>                          spoof)
>>> print(result)

Examples

>>> from utool.util_grabdata import *  # NOQA
>>> zipped_url = 'https://dl.dropboxusercontent.com/s/of2s82ed4xf86m6/testdata.zip'
>>> zipped_url = 'http://www.spam.com/eggs/data.zip'
utool.util_grabdata.open_url_in_browser(url, browsername=None, fallback=False)[source]

Opens a url in the specified or default browser

Parameters:url (str) – web url

CommandLine:

python -m utool.util_grabdata --test-open_url_in_browser

Example

>>> # SCRIPT
>>> from utool.util_grabdata import *  # NOQA
>>> url = 'http://www.jrsoftware.org/isdl.php'
>>> open_url_in_browser(url, 'chrome')
utool.util_grabdata.read_s3_contents(bucket, key, auth_access_id=None, auth_secret_key=None, auth_domain=None)[source]
utool.util_grabdata.rsync(src_uri, dst_uri, exclude_dirs=[], port=22, dryrun=False)[source]

Wrapper for rsync

General function to push or pull a directory from a remote server to a local path

References

http://www.tecmint.com/rsync-local-remote-file-synchronization-commands/ http://serverfault.com/questions/219013/show-progress-in-rsync

Notes (rsync commandline options):

rsync [OPTION]... SRC [SRC]... DEST -v : verbose -r : copies data recursively (but dont preserve timestamps and

permission while transferring data
-a : archive mode, allows recursive copying and preserves symlinks,
permissions, user and group ownerships, and timestamps

-z : compress file data -i, –itemize-changes output a change-summary for all updates -s, –protect-args : no space-splitting; only wildcard special-chars -h : human-readable, output numbers in a human-readable format -P same as –partial –progress

utool.util_grabdata.s3_dict_encode_to_str(s3_dict)[source]
utool.util_grabdata.s3_str_decode_to_dict(s3_str)[source]
utool.util_grabdata.scp_pull(remote_path, local_path='.', remote='localhost', user=None)[source]

wrapper for scp

utool.util_grabdata.split_archive_ext(path)[source]
utool.util_grabdata.unarchive_file(archive_fpath, force_commonprefix=True)[source]
utool.util_grabdata.untar_file(targz_fpath, force_commonprefix=True)[source]
utool.util_grabdata.unzip_file(zip_fpath, force_commonprefix=True, output_dir=None, prefix=None, dryrun=False, overwrite=None)[source]
utool.util_grabdata.url_read(url, verbose=True)[source]

Directly reads data from url

utool.util_graph module

utool.util_graph.dfs_template(graph, previsit, postvisit)[source]
utool.util_graph.dict_depth(dict_, accum=0)[source]
utool.util_graph.edges_to_adjacency_list(edges)[source]
utool.util_graph.find_odd_cycle()[source]

given any starting point in an scc if there is an odd length cycle in the scc then the starting node is part of the odd length cycle

Let s* be part of the odd length cycle Start from any point s There is also a cycle from (s* to s) due to scc. If that cycle is even, then go to s*, then go in the odd length cycle back to s* and then go back to s, which makes this path odd. If s s* s is odd we are done.

because it is strongly connected there is a path from s* to s and s to s*. If that cycle is odd then done otherwise,

# Run pairity check on each scc # Then check all edges for equal pairity

CommandLine:

python -m utool.util_graph --exec-find_odd_cycle --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> result = find_odd_cycle()
>>> print(result)
>>> ut.show_if_requested()
utool.util_graph.get_allkeys(dict_)[source]
utool.util_graph.get_ancestor_levels(graph, tablename)[source]
utool.util_graph.get_descendant_levels(graph, tablename)[source]
utool.util_graph.get_levels(dict_, n=0, levels=None)[source]
Parameters:
  • dict (dict_) – a dictionary
  • n (int) – (default = 0)
  • levels (None) – (default = None)

CommandLine:

python -m utool.util_graph --test-get_levels --show
python3 -m utool.util_graph --test-get_levels --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> from_root = {
>>>     'dummy_annot': {
>>>         'chip': {
>>>                 'keypoint': {
>>>                             'fgweight': None,
>>>                         },
>>>             },
>>>         'probchip': {
>>>                 'fgweight': None,
>>>             },
>>>     },
>>> }
>>> dict_ = from_root
>>> n = 0
>>> levels = None
>>> levels_ = get_levels(dict_, n, levels)
>>> result = ut.repr2(levels_, nl=1)
>>> print(result)
[
    ['dummy_annot'],
    ['chip', 'probchip'],
    ['keypoint', 'fgweight'],
    ['fgweight'],
]
utool.util_graph.longest_levels(levels_)[source]
Parameters:levels (list) –

CommandLine:

python -m utool.util_graph --exec-longest_levels --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> levels_ = [
>>>     ['dummy_annot'],
>>>     ['chip', 'probchip'],
>>>     ['keypoint', 'fgweight'],
>>>     ['fgweight'],
>>> ]
>>> new_levels = longest_levels(levels_)
>>> result = ('new_levels = %s' % (ut.repr2(new_levels, nl=1),))
>>> print(result)
new_levels = [
    ['dummy_annot'],
    ['chip', 'probchip'],
    ['keypoint'],
    ['fgweight'],
]
utool.util_graph.path_to_leafs(tablename, parent_to_children)[source]
utool.util_graph.paths_to_root(tablename, root, child_to_parents)[source]

CommandLine:

python -m utool.util_graph --exec-paths_to_root:0
python -m utool.util_graph --exec-paths_to_root:1

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> child_to_parents = {
>>>     'chip': ['dummy_annot'],
>>>     'chipmask': ['dummy_annot'],
>>>     'descriptor': ['keypoint'],
>>>     'fgweight': ['keypoint', 'probchip'],
>>>     'keypoint': ['chip'],
>>>     'notch': ['dummy_annot'],
>>>     'probchip': ['dummy_annot'],
>>>     'spam': ['fgweight', 'chip', 'keypoint']
>>> }
>>> root = 'dummy_annot'
>>> tablename = 'fgweight'
>>> to_root = paths_to_root(tablename, root, child_to_parents)
>>> result = ut.repr3(to_root)
>>> print(result)
{
    'keypoint': {
        'chip': {
                'dummy_annot': None,
            },
    },
    'probchip': {
        'dummy_annot': None,
    },
}

Example

>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> root = u'annotations'
>>> tablename = u'Notch_Tips'
>>> child_to_parents = {
>>>     'Block_Curvature': [
>>>         'Trailing_Edge',
>>>     ],
>>>     'Has_Notch': [
>>>         'annotations',
>>>     ],
>>>     'Notch_Tips': [
>>>         'annotations',
>>>     ],
>>>     'Trailing_Edge': [
>>>         'Notch_Tips',
>>>     ],
>>> }
>>> to_root = paths_to_root(tablename, root, child_to_parents)
>>> result = ut.repr3(to_root)
>>> print(result)
utool.util_graph.reverse_path(dict_, root, child_to_parents)[source]

CommandLine:

python -m utool.util_graph --exec-reverse_path --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> child_to_parents = {
>>>     'chip': ['dummy_annot'],
>>>     'chipmask': ['dummy_annot'],
>>>     'descriptor': ['keypoint'],
>>>     'fgweight': ['keypoint', 'probchip'],
>>>     'keypoint': ['chip'],
>>>     'notch': ['dummy_annot'],
>>>     'probchip': ['dummy_annot'],
>>>     'spam': ['fgweight', 'chip', 'keypoint']
>>> }
>>> to_root = {
>>>     'fgweight': {
>>>         'keypoint': {
>>>                 'chip': {
>>>                             'dummy_annot': None,
>>>                         },
>>>             },
>>>         'probchip': {
>>>                 'dummy_annot': None,
>>>             },
>>>     },
>>> }
>>> reversed_ = reverse_path(to_root, 'dummy_annot', child_to_parents)
>>> result = ut.repr3(reversed_)
>>> print(result)
{
    'dummy_annot': {
        'chip': {
                'keypoint': {
                            'fgweight': None,
                        },
            },
        'probchip': {
                'fgweight': None,
            },
    },
}
utool.util_graph.shortest_levels(levels_)[source]
Parameters:levels (list) –

CommandLine:

python -m utool.util_graph --exec-shortest_levels --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> levels_ = [
>>>     ['dummy_annot'],
>>>     ['chip', 'probchip'],
>>>     ['keypoint', 'fgweight'],
>>>     ['fgweight'],
>>> ]
>>> new_levels = shortest_levels(levels_)
>>> result = ('new_levels = %s' % (ut.repr2(new_levels, nl=1),))
>>> print(result)
new_levels = [
    ['dummy_annot'],
    ['chip', 'probchip'],
    ['keypoint', 'fgweight'],
]
utool.util_graph.testdata_graph()[source]
Returns:(graph, G)
Return type:tuple

CommandLine:

python -m utool.util_graph --exec-testdata_graph --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_graph import *  # NOQA
>>> import utool as ut
>>> (graph, G) = testdata_graph()
>>> import plottool as pt
>>> ut.ensure_pylab_qt4()
>>> pt.show_nx(G, layout='pygraphviz')
>>> ut.show_if_requested()
utool.util_graph.topsort_dfs(G, previsit, postvisit)[source]
utool.util_graph.traverse_path(start, end, seen_, allkeys, mat)[source]

utool.util_gridsearch module

module for gridsearch helper

class utool.util_gridsearch.DimensionBasis(dimension_name, dimension_point_list)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__getstate__()

Exclude the OrderedDict from pickling

__repr__()

Return a nicely formatted representation string

dimension_name

Alias for field number 0

dimension_point_list

Alias for field number 1

class utool.util_gridsearch.GridSearch(gridsearch, grid_basis, label=None)[source]

Bases: object

helper for executing iterations and analyzing the results of a grid search

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> grid_basis = [
...     ut.DimensionBasis('p', [.5, .8, .9, 1.0]),
...     ut.DimensionBasis('K', [2, 3, 4, 5]),
...     ut.DimensionBasis('dcvs_clip_max', [.1, .2, .5, 1.0]),
... ]
>>> gridsearch = ut.GridSearch(grid_basis, label='testdata_gridsearch')
>>> for cfgdict in gridsearch:
...    tp_score = cfgdict['p'] + (cfgdict['K'] ** .5)
...    tn_score = (cfgdict['p'] * (cfgdict['K'])) / cfgdict['dcvs_clip_max']
...    gridsearch.append_result(tp_score, tn_score)
append_result(gridsearch, tp_score, tn_score)[source]

for use in iteration

get_csv_results(gridsearch, max_lines=None, score_lbl=u'score_diff')[source]

Make csv text describing results

Parameters:
  • max_lines (int) – add top num lines to the csv. No limit if None.
  • score_lbl (str) – score label to sort by
Returns:

result data in csv format
Return type:

str

CommandLine:

python -m utool.util_gridsearch --test-get_csv_results

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> import plottool as pt
>>> # build test data
>>> score_lbl = 'score_diff'
>>> gridsearch = testdata_grid_search()
>>> csvtext = gridsearch.get_csv_results(10, score_lbl)
>>> print(csvtext)
>>> result = ut.hashstr(csvtext)
>>> print(result)
60yptleiwo@lk@24
get_dimension_stats(gridsearch, param_lbl, score_lbl=u'score_diff')[source]

Returns result stats about a specific parameter

Parameters:
  • param_lbl (str) –
  • score_lbl (str) – score label to sort by
Returns:

param2_score_stats
Return type:

dict
get_dimension_stats_str(gridsearch, param_lbl, score_lbl=u'score_diff')[source]

Returns a result stat string about a specific parameter

get_param_lbls(gridsearch, exclude_unvaried_dimension=True)[source]
get_param_list_and_lbls(gridsearch)[source]

returns input data

get_rank_cfgdict(gridsearch, rank=0, score_lbl=u'score_diff')[source]
get_score_list_and_lbls(gridsearch)[source]

returns result data

get_sorted_columns_and_labels(gridsearch, score_lbl=u'score_diff')[source]

returns sorted input and result data

plot_dimension(gridsearch, param_lbl, score_lbl=u'score_diff', **kwargs)[source]

Plots result statistics about a specific parameter

Parameters:
  • param_lbl (str) –
  • score_lbl (str) –

CommandLine:

python -m utool.util_gridsearch --test-plot_dimension
python -m utool.util_gridsearch --test-plot_dimension --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import plottool as pt
>>> # build test data
>>> gridsearch = testdata_grid_search()
>>> param_lbl = 'p'
>>> score_lbl = 'score_diff'
>>> self = gridsearch
>>> self.plot_dimension('p', score_lbl, fnum=1, pnum=(1, 3, 1))
>>> self.plot_dimension('K', score_lbl, fnum=1, pnum=(1, 3, 2))
>>> self.plot_dimension('dcvs_clip_max', score_lbl, fnum=1, pnum=(1, 3, 3))
>>> pt.show_if_requested()
rrr(verbose=True)

special class reloading function

class utool.util_gridsearch.ParamInfo(pi, varname, default, shortprefix=<utool.util_const.ClassNoParam object>, type_=<utool.util_const.ClassNoParam object>, varyvals=[], varyslice=None, hideif=<utool.util_const.ClassNoParam object>, help_=None, valid_values=None)[source]

Bases: utool.util_dev.NiceRepr

small class for individual paramater information

Configuration objects should use these for default / printing / type information however, the actual value of the parameter for any specific configuration is not stored here.

append_hideif(pi, hideif)[source]
get_itemstr(pi, cfg)[source]
is_hidden(pi, cfg)[source]
make_itemstr(pi, cfg)[source]
make_varstr(pi, varval)[source]
rrr(verbose=True)

special class reloading function

class utool.util_gridsearch.ParamInfoBool(pi, varname, default=False, shortprefix=<utool.util_const.ClassNoParam object>, type_=<type 'bool'>, varyvals=[], varyslice=None, hideif=False, help_=None)[source]

Bases: utool.util_gridsearch.ParamInfo

param info for booleans

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> pi = ParamInfoBool('cheese_on', hideif=util_dev.NoParam)
>>> cfg = ut.DynStruct()
>>> cfg.cheese_on = False
>>> result = pi.get_itemstr(cfg)
>>> print(result)
nocheese
make_itemstr(pi, cfg)[source]
rrr(verbose=True)

special class reloading function

class utool.util_gridsearch.ParamInfoList(name, param_info_list=[], constraint_func=None, hideif=None)[source]

Bases: object

small class for ut.Pref-less configurations

append_hideif(hideif)[source]
aslist(hideif=None)[source]
get_grid_basis()[source]

DEPRICATE

get_gridsearch_input(defaultslice=slice(0, 1, None))[source]

for gridsearch

get_slicedict()[source]

for gridsearch

get_varnames()[source]
get_varydict()[source]

for gridsearch

rrr(verbose=True)

special class reloading function

updated_cfgdict(dict_)[source]
utool.util_gridsearch.constrain_cfgdict_list(cfgdict_list_, constraint_func)[source]

constrains configurations and removes duplicates

utool.util_gridsearch.customize_base_cfg(cfgname, cfgopt_strs, base_cfg, cfgtype, alias_keys=None, valid_keys=None, offset=0, strict=True)[source]
Parameters:
  • cfgname (str) – config name
  • cfgopt_strs (str) – mini-language defining key variations
  • base_cfg (dict) – specifies the default cfg to customize
  • cfgtype –
  • alias_keys (None) – (default = None)
  • valid_keys (None) – if base_cfg is not specied, this defines the valid keys (default = None)
  • offset (int) – (default = 0)
  • strict (bool) – (default = True)
Returns:

cfg_combo - list of config dicts defining customized configs based on cfgopt_strs. customized configs always are given an _cfgindex, _cfgstr, and _cfgname key.
Return type:

list

CommandLine:

python -m utool.util_gridsearch --exec-_customize_base_cfg

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfgname = 'default'
>>> cfgopt_strs = 'dsize=1000,per_name=[1,2]'
>>> base_cfg = '?'
>>> cfgtype = '?'
>>> alias_keys = None
>>> valid_keys = None
>>> offset = 0
>>> strict = True
>>> cfg_combo = customize_base_cfg(cfgname, cfgopt_strs, base_cfg, cfgtype,
>>>                                alias_keys, valid_keys, offset, strict)
>>> result = ('cfg_combo = %s' % (cfg_combo,))
>>> print(result)
utool.util_gridsearch.get_cfg_lbl(cfg, name=None, nonlbl_keys=[u'_cfgstr', u'_cfgname', u'_cfgtype', u'_cfgindex'], key_order=None)[source]

Formats a flat configuration dict into a short string label

Parameters:
  • cfg (dict) –
  • name (str) – (default = None)
  • nonlbl_keys (list) – (default = INTERNAL_CFGKEYS)
Returns:

cfg_lbl
Return type:

str

CommandLine:

python -m utool.util_gridsearch --exec-get_cfg_lbl

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfg = {'_cfgname': 'test', 'var1': 'val1', 'var2': 'val2'}
>>> name = None
>>> nonlbl_keys = ['_cfgstr', '_cfgname', '_cfgtype', '_cfgindex']
>>> cfg_lbl = get_cfg_lbl(cfg, name, nonlbl_keys)
>>> result = ('cfg_lbl = %s' % (six.text_type(cfg_lbl),))
>>> print(result)
cfg_lbl = test:var1=val1,var2=val2

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfg = {'_cfgname': 'test:K=[1,2,3]', 'K': '1'}
>>> name = None
>>> nonlbl_keys = ['_cfgstr', '_cfgname', '_cfgtype', '_cfgindex']
>>> cfg_lbl = get_cfg_lbl(cfg, name, nonlbl_keys)
>>> result = ('cfg_lbl = %s' % (six.text_type(cfg_lbl),))
>>> print(result)
cfg_lbl = test:K=1
utool.util_gridsearch.get_cfgdict_lbl_list_subset(cfgdict_list, varied_dict)[source]
utool.util_gridsearch.get_cfgdict_list_subset(cfgdict_list, keys)[source]

returns list of unique dictionaries only with keys specified in keys

Parameters:
  • cfgdict_list (list) –
  • keys (list) –
Returns:

cfglbl_list
Return type:

list

CommandLine:

python -m utool.util_gridsearch --test-get_cfgdict_list_subset

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> cfgdict_list = [
...    {'K': 3, 'dcvs_clip_max': 0.1, 'p': 0.1},
...    {'K': 5, 'dcvs_clip_max': 0.1, 'p': 0.1},
...    {'K': 5, 'dcvs_clip_max': 0.1, 'p': 0.2},
...    {'K': 3, 'dcvs_clip_max': 0.2, 'p': 0.1},
...    {'K': 5, 'dcvs_clip_max': 0.2, 'p': 0.1},
...    {'K': 3, 'dcvs_clip_max': 0.2, 'p': 0.1}]
>>> keys = ['K', 'dcvs_clip_max']
>>> # execute function
>>> cfgdict_sublist = get_cfgdict_list_subset(cfgdict_list, keys)
>>> # verify results
>>> result = ut.list_str(cfgdict_sublist)
>>> print(result)
[
    {'K': 3, 'dcvs_clip_max': 0.1},
    {'K': 5, 'dcvs_clip_max': 0.1},
    {'K': 3, 'dcvs_clip_max': 0.2},
    {'K': 5, 'dcvs_clip_max': 0.2},
]
utool.util_gridsearch.get_nonvaried_cfg_lbls(cfg_list, default_cfg=None, mainkey=u'_cfgname')[source]
TODO: this might only need to return a single value. Maybe not if the names
are different.
Parameters:
  • cfg_list (list) –
  • default_cfg (None) – (default = None)
Returns:

cfglbl_list
Return type:

list

CommandLine:

python -m utool.util_gridsearch --exec-get_nonvaried_cfg_lbls

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfg_list = [{'_cfgname': 'test', 'f': 1, 'b': 1},
>>>             {'_cfgname': 'test', 'f': 2, 'b': 1},
>>>             {'_cfgname': 'test', 'f': 3, 'b': 1, 'z': 4}]
>>> default_cfg = None
>>> cfglbl_list = get_nonvaried_cfg_lbls(cfg_list, default_cfg)
>>> result = ('cfglbl_list = %s' % (ut.repr2(cfglbl_list),))
>>> print(result)
cfglbl_list = ['test:b=1', 'test:b=1', 'test:b=1']
utool.util_gridsearch.get_varied_cfg_lbls(cfg_list, default_cfg=None, mainkey=u'_cfgname')[source]
Parameters:
  • cfg_list (list) –
  • default_cfg (None) – (default = None)
Returns:

cfglbl_list
Return type:

list

CommandLine:

python -m utool.util_gridsearch --exec-get_varied_cfg_lbls

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfg_list = [{'_cfgname': 'test', 'f': 1, 'b': 1},
>>>             {'_cfgname': 'test', 'f': 2, 'b': 1},
>>>             {'_cfgname': 'test', 'f': 3, 'b': 1, 'z': 4}]
>>> default_cfg = None
>>> cfglbl_list = get_varied_cfg_lbls(cfg_list, default_cfg)
>>> result = ('cfglbl_list = %s' % (ut.repr2(cfglbl_list),))
>>> print(result)
cfglbl_list = ['test:f=1', 'test:f=2', 'test:f=3,z=4']
utool.util_gridsearch.grid_search_generator(grid_basis=[], *args, **kwargs)[source]

Iteratively yeilds individual configuration points inside a defined basis.

Parameters:grid_basis (list) – a list of 2-component tuple. The named tuple looks like this:

CommandLine:

python -m utool.util_gridsearch --test-grid_search_generator

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> grid_basis = [
... DimensionBasis('dim1', [.1, .2, .3]),
... DimensionBasis('dim2', [.1, .4, .5]),
... ]
>>> args = tuple()
>>> kwargs = {}
>>> # execute function
>>> point_list = list(grid_search_generator(grid_basis))
>>> # verify results
>>> column_lbls = ut.get_list_column(grid_basis, 0)
>>> column_list  = ut.get_list_column(grid_basis, 1)
>>> first_vals = ut.get_list_column(ut.get_list_column(grid_basis, 1), 0)
>>> column_types = list(map(type, first_vals))
>>> header = 'grid search'
>>> result = ut.make_csv_table(column_list, column_lbls, header, column_types)
>>> print(result)
grid search
# num_rows=3
#   dim1,  dim2
    0.10,  0.10
    0.20,  0.40
    0.30,  0.50
utool.util_gridsearch.gridsearch_timer(func_list, args_list, niters=None, **searchkw)[source]

Times a series of functions on a series of inputs

args_list is a list should vary the input sizes can also be a func that take a count param

items in args_list list or returned by the func should be a tuple so it can be unpacked

CommandLine:

python -m ibeis.annotmatch_funcs --exec-get_annotmatch_rowids_from_aid2 --show
python -m ibeis.annotmatch_funcs --exec-get_annotmatch_rowids_from_aid:1 --show
Parameters:
  • func_list (list) –
  • args_list (list) –
  • niters (None) – (default = None)
Returns:

time_result
Return type:

dict

CommandLine:

python -m utool.util_gridsearch --exec-gridsearch_timer --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> func_list = [ut.fibonacci_recursive, ut.fibonacci_iterative]
>>> args_list = list(range(1, 35))
>>> niters = None
>>> searchkw = {}
>>> time_result = gridsearch_timer(func_list, args_list, niters, **searchkw)
>>> result = ('time_result = %s' % (six.text_type(time_result),))
>>> print(result)
>>> time_result['plot_timings']()
>>> ut.show_if_requested()
utool.util_gridsearch.interact_gridsearch_result_images(show_result_func, cfgdict_list, cfglbl_list, cfgresult_list, score_list=None, fnum=None, figtitle=u'', unpack=False, max_plots=25, verbose=True, precision=3, scorelbl=u'score', onclick_func=None)[source]

helper function for visualizing results of gridsearch

utool.util_gridsearch.lookup_base_cfg_list(cfgname, named_defaults_dict, metadata=None)[source]
utool.util_gridsearch.make_cfglbls(cfgdict_list, varied_dict)[source]

Show only the text in labels that mater from the cfgdict

utool.util_gridsearch.make_constrained_cfg_and_lbl_list(varied_dict, constraint_func=None, slice_dict=None, defaultslice=slice(0, 1, None))[source]
Parameters:
  • varied_dict (dict) – parameters to vary with possible variations
  • constraint_func (func) – function to restirct parameter variations
  • slice_dict (dict) – dict of slices for each param of valid possible values
  • defaultslice (slice) – default slice used if slice is not specified in slice_dict
Returns:

(cfgdict_list, cfglbl_list)
Return type:

tuple

CommandLine:

python -m utool.util_gridsearch --test-make_constrained_cfg_and_lbl_list

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> # build test data
>>> varied_dict = {
...     'p': [.1, .3, 1.0, 2.0],
...     'dcvs_clip_max': [.1, .2, .5],
...     'K': [3, 5],
... }
>>> constraint_func = None
>>> # execute function
>>> (cfgdict_list, cfglbl_list) = make_constrained_cfg_and_lbl_list(varied_dict, constraint_func)
>>> # verify results
>>> result = six.text_type((cfgdict_list, cfglbl_list))
>>> print(result)
utool.util_gridsearch.noexpand_parse_cfgstrs(cfgopt_strs, alias_keys=None)[source]
utool.util_gridsearch.parse_argv_cfg(argname, default=[u''], named_defaults_dict=None, valid_keys=None, alias_keys=None)[source]

simple configs

Parameters:
  • argname (str) –
  • default (list) – (default = [])
  • named_defaults_dict (dict) – (default = None)
  • valid_keys (None) – (default = None)
Returns:

cfg_list
Return type:

list

CommandLine:

python -m utool.util_gridsearch --exec-parse_argv_cfg --filt :foo=bar
python -m utool.util_gridsearch --exec-parse_argv_cfg

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> argname = '--filt'
>>> cfg_list = parse_argv_cfg(argname)
>>> result = ('cfg_list = %s' % (six.text_type(cfg_list),))
>>> print(result)
utool.util_gridsearch.parse_cfgstr_list2(cfgstr_list, named_defaults_dict=None, cfgtype=None, alias_keys=None, valid_keys=None, expand_nested=True, strict=True, special_join_dict=None, is_nestedcfgtype=False, metadata=None)[source]

Parses config strings. By looking up name in a dict of configs

Parameters:
  • cfgstr_list (list) –
  • named_defaults_dict (dict) – (default = None)
  • cfgtype (None) – (default = None)
  • alias_keys (None) – (default = None)
  • valid_keys (None) – (default = None)
  • expand_nested (bool) – (default = True)
  • strict (bool) – (default = True)
  • - used for annot configs so special joins arent geometrically combined (is_nestedcfgtype) –

Note

Normal Case:
–flag name
Custom Arugment Cases:
–flag name:custom_key1=custom_val1,custom_key2=custom_val2
Multiple Config Case:
–flag name1:custom_args1 name2:custom_args2
Multiple Config (special join) Case:
(here name2 and name3 have some special interaction) –flag name1:custom_args1 name2:custom_args2::name3:custom_args3
Varied Argument Case:
–flag name:key1=[val1,val2]
Returns:cfg_combos_list
Return type:list

CommandLine:

python -m utool.util_gridsearch --exec-parse_cfgstr_list2
python -m utool.util_gridsearch --exec-parse_cfgstr_list2:0
python -m utool.util_gridsearch --exec-parse_cfgstr_list2:1
Setup:
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> named_defaults_dict = None
>>> cfgtype = None
>>> alias_keys = None
>>> valid_keys = None
>>> expand_nested = True
>>> strict = False
>>> metadata = None
>>> is_nestedcfgtype=False
Example0:
>>> # ENABLE_DOCTEST
>>> cfgstr_list = ['name', 'name:f=1', 'name:b=[1,2]', 'name1:f=1::name2:f=1,b=2']
>>> #cfgstr_list = ['name', 'name1:f=1::name2:f=1,b=2']
>>> special_join_dict = {'joined': True}
>>> cfg_combos_list = parse_cfgstr_list2(
>>>     cfgstr_list, named_defaults_dict, cfgtype, alias_keys, valid_keys,
>>>     expand_nested, strict, special_join_dict)
>>> print('cfg_combos_list = %s' % (ut.list_str(cfg_combos_list, nl=2),))
>>> print(ut.depth_profile(cfg_combos_list))
>>> cfg_list = ut.flatten(cfg_combos_list)
>>> cfg_list = ut.flatten([cfg if isinstance(cfg, list) else [cfg] for cfg in cfg_list])
>>> result = ut.repr2(ut.get_varied_cfg_lbls(cfg_list))
>>> print(result)
['name:', 'name:f=1', 'name:b=1', 'name:b=2', 'name1:f=1,joined=True', 'name2:b=2,f=1,joined=True']
Example1:
>>> # ENABLE_DOCTEST
>>> # Allow for definition of a named default on the fly
>>> cfgstr_list = ['base=:f=2,c=[1,2]', 'base:f=1', 'base:b=[1,2]']
>>> special_join_dict = None
>>> cfg_combos_list = parse_cfgstr_list2(
>>>     cfgstr_list, named_defaults_dict, cfgtype, alias_keys, valid_keys,
>>>     expand_nested, strict, special_join_dict)
>>> print('cfg_combos_list = %s' % (ut.list_str(cfg_combos_list, nl=2),))
>>> print(ut.depth_profile(cfg_combos_list))
>>> cfg_list = ut.flatten(cfg_combos_list)
>>> cfg_list = ut.flatten([cfg if isinstance(cfg, list) else [cfg] for cfg in cfg_list])
>>> result = ut.repr2(ut.get_varied_cfg_lbls(cfg_list))
>>> print(result)
['base:c=1,f=1', 'base:c=2,f=1', 'base:b=1,c=1,f=2', 'base:b=1,c=2,f=2', 'base:b=2,c=1,f=2', 'base:b=2,c=2,f=2']
utool.util_gridsearch.parse_cfgstr_name_options(cfgstr)[source]
Parameters:cfgstr (str) –
Returns:(cfgname, cfgopt_strs, subx)
Return type:tuple

CommandLine:

python -m utool.util_gridsearch --test-parse_cfgstr_name_options

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfgstr = 'default' + NAMEVARSEP + 'myvar1=myval1,myvar2=myval2'
>>> (cfgname, cfgopt_strs, subx) = parse_cfgstr_name_options(cfgstr)
>>> result = ('(cfgname, cfg_optstrs, subx) = %s' % (ut.repr2((cfgname, cfgopt_strs, subx)),))
>>> print(result)
(cfgname, cfg_optstrs, subx) = ('default', 'myvar1=myval1,myvar2=myval2', None)

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfgstr = 'default[0:1]' + NAMEVARSEP + 'myvar1=myval1,myvar2=myval2'
>>> (cfgname, cfgopt_strs, subx) = parse_cfgstr_name_options(cfgstr)
>>> result = ('(cfgname, cfg_optstrs, subx) = %s' % (ut.repr2((cfgname, cfgopt_strs, subx)),))
>>> print(result)
(cfgname, cfg_optstrs, subx) = ('default', 'myvar1=myval1,myvar2=myval2', slice(0, 1, None))

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfgstr = 'default[0]' + NAMEVARSEP + 'myvar1=myval1,myvar2=myval2'
>>> (cfgname, cfgopt_strs, subx) = parse_cfgstr_name_options(cfgstr)
>>> result = ('(cfgname, cfg_optstrs, subx) = %s' % (ut.repr2((cfgname, cfgopt_strs, subx)),))
>>> print(result)
(cfgname, cfg_optstrs, subx) = ('default', 'myvar1=myval1,myvar2=myval2', [0])
utool.util_gridsearch.partition_varied_cfg_list(cfg_list, default_cfg=None, recursive=False)[source]

TODO: partition nested configs

CommandLine:

python -m utool.util_gridsearch --exec-partition_varied_cfg_list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfg_list = [{'f': 1, 'b': 1}, {'f': 2, 'b': 1}, {'f': 3, 'b': 1, 'z': 4}]
>>> nonvaried_cfg, varied_cfg_list = partition_varied_cfg_list(cfg_list)
>>> result = ut.list_str((nonvaried_cfg, varied_cfg_list), label_list=['nonvaried_cfg', 'varied_cfg_list'])
>>> print(result)
nonvaried_cfg = {'b': 1}
varied_cfg_list = [{'f': 1}, {'f': 2}, {'f': 3, 'z': 4}]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_gridsearch import *  # NOQA
>>> import utool as ut
>>> cfg_list = [{'q1': 1, 'f1': {'a2': {'x3': 1, 'y3': 2}, 'b2': 1}}, {'q1': 1, 'f1': {'a2': {'x3': 1, 'y3':1}, 'b2': 1}, 'e1': 1}]
>>> print(ut.list_str(cfg_list, nl=True))
>>> nonvaried_cfg, varied_cfg_list = partition_varied_cfg_list(cfg_list, recursive=True)
>>> result = ut.list_str((nonvaried_cfg, varied_cfg_list), label_list=['nonvaried_cfg', 'varied_cfg_list'])
>>> print(result)
nonvaried_cfg = {'f1': {'a2': {'x3': 1}, 'b2': 1}, 'q1': 1}
varied_cfg_list = [{'f1': {'a2': {'y3': 2}}}, {'e1': 1, 'f1': {'a2': {'y3': 1}}}]

test data function for doctests

utool.util_hash module

Hashing convinience functions

You should opt to use a hash*27 function over a hash* function.

TODO: the same hashing algorithm should be used everywhere Currently there is a mix of sha1, sha256, and sha512 in different places.

utool.util_hash.augment_uuid(uuid_, *hashables)[source]
utool.util_hash.convert_hexstr_to_bigbase(hexstr, alphabet=[u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f', u'g', u'h', u'i', u'j', u'k', u'l', u'm', u'n', u'o', u'p', u'q', u'r', u's', u't', u'u', u'v', u'w', u'x', u'y', u'z', u'@', u'!', u'%', u'&', u'+'], bigbase=41)[source]

Packs a long hexstr into a shorter length string with a larger base

utool.util_hash.deterministic_uuid(hashable)[source]
utool.util_hash.get_file_hash(fpath, blocksize=65536, hasher=None, stride=1)[source]

For better hashes use hasher=hashlib.sha256, and keep stride=1

Parameters:
  • fpath (str) – file path string
  • blocksize (int) – 2 ** 16. Affects speed of reading file
  • hasher (None) – defaults to sha1 for fast (but insecure) hashing
  • stride (int) – strides > 1 skip data to hash, useful for faster hashing, but less accurate, also makes hash dependant on blocksize.

References

http://stackoverflow.com/questions/3431825/generating-a-md5-checksum-of-a-file http://stackoverflow.com/questions/5001893/when-should-i-use-sha-1-and-when-should-i-use-sha-2

CommandLine:

python -m utool.util_hash --test-get_file_hash
python -m utool.util_hash --test-get_file_hash:0
python -m utool.util_hash --test-get_file_hash:1

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> fpath = ut.grab_test_imgpath('patsy.jpg')
>>> #blocksize = 65536  # 2 ** 16
>>> blocksize = 2 ** 16
>>> hasher = None
>>> stride = 1
>>> hashbytes_20 = get_file_hash(fpath, blocksize, hasher, stride)
>>> result = repr(hashbytes_20)
>>> print(result)
'7\x07B\x0eX<sRu\xa2\x90P\xda\xb2\x84?\x81?\xa9\xd9'

‘x13x9bxf6x0fxa3QQ xd7”$xe9mx05x9ex81xf6xf2vxe4’

‘x16x00x80Xxx8c-HxcdPxf6x02x9frlxbfx99VQxb5’

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> #fpath = ut.grab_file_url('http://en.wikipedia.org/wiki/List_of_comets_by_type')
>>> fpath = ut.unixjoin(ut.ensure_app_resource_dir('utool'), 'tmp.txt')
>>> ut.write_to(fpath, ut.lorium_ipsum())
>>> blocksize = 2 ** 3
>>> hasher = None
>>> stride = 2
>>> hashbytes_20 = get_file_hash(fpath, blocksize, hasher, stride)
>>> result = repr(hashbytes_20)
>>> print(result)
'5KP\xcf>R\xf6\xffO:L\xac\x9c\xd3V+\x0e\xf6\xe1n'
utool.util_hash.get_file_uuid(fpath, hasher=None, stride=1)[source]

Creates a uuid from the hash of a file

utool.util_hash.get_zero_uuid()[source]
utool.util_hash.hashable_to_uuid(hashable_)[source]

TODO: ensure that python2 and python3 agree on hashes of the same information

Parameters:hashable (hashable) – hashables are bytes-like objects An object that supports the Buffer Protocol, like bytes, bytearray or memoryview. Bytes-like objects can be used for various operations that expect binary data, such as compression, saving to a binary file or sending over a socket. Some operations need the binary data to be mutable, in which case not all bytes-like objects can apply.
Returns:uuid_
Return type:UUID

CommandLine:

python -m utool.util_hash --test-hashable_to_uuid
python3 -m utool.util_hash --test-hashable_to_uuid:0
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> hashable_ = 'foobar'
>>> uuid_ = hashable_to_uuid(hashable_)
>>> result = str(uuid_)
>>> print(result)
8843d7f9-2416-211d-e9eb-b963ff4ce281
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> hashable_ = u'foobar'
>>> uuid_ = hashable_to_uuid(hashable_)
>>> result = str(uuid_)
>>> print(result)
8843d7f9-2416-211d-e9eb-b963ff4ce281
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> hashable_ = 10
>>> uuid_ = hashable_to_uuid(hashable_)
>>> result = str(uuid_)
>>> print(result)
b1d57811-11d8-4f7b-3fe4-5a0852e59758
utool.util_hash.hashstr(data, hashlen=16, alphabet=[u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f', u'g', u'h', u'i', u'j', u'k', u'l', u'm', u'n', u'o', u'p', u'q', u'r', u's', u't', u'u', u'v', u'w', u'x', u'y', u'z', u'@', u'!', u'%', u'&', u'+'])[source]

python -c “import utool as ut; print(ut.hashstr(‘abcd’))”

Parameters:
  • data (hashable) –
  • hashlen (int) – (default = 16)
  • alphabet (list) – list of characters:
Returns:

hashstr
Return type:

str

CommandLine:

python -m utool.util_hash --test-hashstr
python3 -m utool.util_hash --test-hashstr

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> data = 'foobar'
>>> hashlen = 16
>>> alphabet = ALPHABET
>>> hashstr = hashstr(data, hashlen, alphabet)
>>> result = ('hashstr = %s' % (str(hashstr),))
>>> print(result)
hashstr = mi5yum60mbxhyp+x

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> data = ''
>>> hashlen = 16
>>> alphabet = ALPHABET
>>> hashstr = hashstr(data, hashlen, alphabet)
>>> result = ('hashstr = %s' % (str(hashstr),))
>>> print(result)
hashstr = 0000000000000000
utool.util_hash.hashstr27(data, alphabet=[u'a', u'b', u'c', u'd', u'e', u'f', u'g', u'h', u'i', u'j', u'k', u'l', u'm', u'n', u'o', u'p', u'q', u'r', u's', u't', u'u', u'v', u'w', u'x', u'y', u'z'], **kwargs)[source]
utool.util_hash.hashstr_arr(arr, lbl=u'arr', pathsafe=False, **kwargs)[source]
Parameters:
  • arr (ndarray) –
  • lbl (str) – (default = ‘arr’)
  • pathsafe (bool) – (default = False)
Returns:

arr_hashstr
Return type:

str

CommandLine:

python -m utool.util_hash --exec-hashstr_arr
python -m utool.util_hash --test-hashstr_arr

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> import numpy as np
>>> arr = np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float64)
>>> lbl = 'arr'
>>> kwargs = {}
>>> pathsafe = False
>>> arr_hashstr = hashstr_arr(arr, lbl, pathsafe, alphabet=ALPHABET_27)
>>> result = ('arr_hashstr = %s' % (str(arr_hashstr),))
>>> print(result)
arr_hashstr = arr((2,3)daukyreqnhfejkfs)
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_hash import *  # NOQA
>>> import numpy as np
>>> arr = np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float64)
>>> kwargs = {}
>>> lbl = 'arr'
>>> pathsafe = True
>>> arr_hashstr = hashstr_arr(arr, lbl, pathsafe, alphabet=ALPHABET_27)
>>> result = ('arr_hashstr = %s' % (str(arr_hashstr),))
>>> print(result)
arr_hashstr = arr-_2,3_daukyreqnhfejkfs-
utool.util_hash.hashstr_arr27(arr, lbl, alphabet=[u'a', u'b', u'c', u'd', u'e', u'f', u'g', u'h', u'i', u'j', u'k', u'l', u'm', u'n', u'o', u'p', u'q', u'r', u's', u't', u'u', u'v', u'w', u'x', u'y', u'z'], **kwargs)[source]
utool.util_hash.hashstr_md5(data)[source]
utool.util_hash.hashstr_sha1(data, base10=False)[source]
utool.util_hash.image_uuid(pil_img)[source]

UNSAFE: DEPRICATE: JPEG IS NOT GAURENTEED TO PRODUCE CONSITENT VALUES ON

MULTIPLE MACHINES image global unique id

References

http://stackoverflow.com/questions/23565889/jpeg-images-have-different-pixel-values-across-multiple-devices

utool.util_hash.make_hash(o)[source]

Makes a hash from a dictionary, list, tuple or set to any level, that contains only other hashable types (including any lists, tuples, sets, and dictionaries). In the case where other kinds of objects (like classes) need to be hashed, pass in a collection of object attributes that are pertinent. For example, a class can be hashed in this fashion:

make_hash([cls.__dict__, cls.__name__])

A function can be hashed like so:

make_hash([fn.__dict__, fn.__code__])

References

http://stackoverflow.com/questions/5884066/hashing-a-python-dictionary

utool.util_hash.random_nonce(length=64, alphabet=None)[source]

returns a random string of len=<length> from <alphabet> I have no idea why this is named random_nonce

utool.util_hash.random_uuid()[source]

utool.util_import module

SeeAlso:
utool._internal.util_importer
class utool.util_import.LazyModule(modname)[source]

Bases: object

Waits to import the module until it is actually used. Caveat: there is no access to module attributes used

in lazy_module_attrs

CommandLine:

python -m utool.util_import --test-LazyModule

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_import import *  # NOQA
>>> import sys
>>> assert 'this' not in sys.modules,  'this was imported before test start'
>>> this = LazyModule('this')
>>> assert 'this' not in sys.modules,  'this should not have been imported yet'
>>> assert this.i == 25
>>> assert 'this' in sys.modules,  'this should now be imported'
>>> print(this)
utool.util_import.import_modname(modname)[source]
Parameters:modname (str) – module name
Returns:module
Return type:module

CommandLine:

python -m utool.util_import --test-import_modname

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_import import *  # NOQA
>>> modname_list = [
>>>     'utool',
>>>     'utool._internal',
>>>     'utool._internal.meta_util_six',
>>>     'utool.util_path',
>>>     #'utool.util_path.checkpath',
>>> ]
>>> modules = [import_modname(modname) for modname in modname_list]
>>> result = ([m.__name__ for m in modules])
>>> assert result == modname_list
utool.util_import.import_module_from_fpath(module_fpath)[source]

imports module from a file path

Args:
module_fpath (str):
Returns:
module: module
CommandLine:
python -m utool.util_import –test-import_module_from_fpath
Example:
>>> # DISABLE_DOCTEST
>>> from utool.util_import import *  # NOQA
>>> module_fpath = '?'
>>> module = import_module_from_fpath(module_fpath)
>>> result = ('module = %s' % (str(module),))
>>> print(result)
Ignore:

module_fpath = ‘/home/joncrall/code/h5py/h5py’ module_fpath = ‘/home/joncrall/code/h5py/build/lib.linux-x86_64-2.7/h5py’ ut.ls(module_fpath) imp.find_module(‘h5py’, ‘/home/joncrall/code/h5py/’)

# Define two temporary modules that are not in sys.path # and have the same name but different values. import sys, os, os.path def ensuredir(path):

if not os.path.exists(path):
os.mkdir(path)

ensuredir(‘tmp’) ensuredir(‘tmp/tmp1’) ensuredir(‘tmp/tmp2’) ensuredir(‘tmp/tmp1/testmod’) ensuredir(‘tmp/tmp2/testmod’) with open(‘tmp/tmp1/testmod/__init__.py’, ‘w’) as file_:

file_.write(‘foo = “spam”
from . import sibling’)
with open(‘tmp/tmp1/testmod/sibling.py’, ‘w’) as file_:
file_.write(‘bar = “ham”’)
with open(‘tmp/tmp2/testmod/__init__.py’, ‘w’) as file_:
file_.write(‘foo = “eggs”
from . import sibling’)
with open(‘tmp/tmp1/testmod/sibling.py’, ‘w’) as file_:
file_.write(‘bar = “jam”’)

# Neither module should be importable through the normal mechanism try:

import testmod assert False, ‘should fail’
except ImportError as ex:
pass

# Try temporarilly adding the directory of a module to the path sys.path.insert(0, ‘tmp/tmp1’) testmod1 = __import__(‘testmod’, globals(), locals(), 0) sys.path.remove(‘tmp/tmp1’) print(testmod1.foo) print(testmod1.sibling.bar)

sys.path.insert(0, ‘tmp/tmp2’) testmod2 = __import__(‘testmod’, globals(), locals(), 0) sys.path.remove(‘tmp/tmp2’) print(testmod2.foo) print(testmod2.sibling.bar)

assert testmod1.foo == “spam” assert testmod1.sibling.bar == “ham”

# Fails, returns spam assert testmod2.foo == “eggs” assert testmod2.sibling.bar == “jam”

sys.path.append(‘/home/username/code/h5py’) import h5py sys.path.append(‘/home/username/code/h5py/build/lib.linux-x86_64-2.7/’) import h5py

utool.util_import.package_contents(package, with_pkg=False, with_mod=True, ignore_prefix=[], ignore_suffix=[])[source]

References

http://stackoverflow.com/questions/1707709/list-all-the-modules-that-are-part-of-a-python-package

Parameters:
  • package –
  • with_pkg (bool) – (default = False)
  • with_mod (bool) – (default = True)

CommandLine:

python -m utool.util_import --exec-package_contents

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_import import *  # NOQA
>>> import ibeis
>>> package = ibeis
>>> ignore_prefix = ['ibeis.tests', 'ibeis.control.__SQLITE3__',
>>>                  '_autogen_explicit_controller']
>>> ignore_suffix = ['_grave']
>>> with_pkg = False
>>> with_mod = True
>>> result = package_contents(package, with_pkg, with_mod,
>>>                           ignore_prefix, ignore_suffix)
>>> print(ut.list_str(result))
utool.util_import.possible_import_patterns(modname)[source]

does not support from x import * does not support from x import z, y

Example

>>> import utool as ut
>>> modname = 'package.submod.submod2.module'
>>> result = ut.repr3(ut.possible_import_patterns(modname))
>>> print(result)
[
    'import package.submod.submod2.module',
    'from package.submod.submod2 import module',
]
utool.util_import.tryimport(modname, pipiname=None, ensure=False)[source]

CommandLine:

python -m utool.util_import --test-tryimport

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_tests import *   # NOQA
>>> import utool as ut
>>> modname = 'pyfiglet'
>>> pipiname = 'git+https://github.com/pwaller/pyfiglet'
>>> pyfiglet = ut.tryimport(modname, pipiname)
>>> assert pyfiglet is None or isinstance(pyfiglet, types.ModuleType), 'unknown error'
Example2:
>>> # UNSTABLE_DOCTEST
>>> # disabled because not everyone has access to being a super user
>>> from utool.util_tests import *   # NOQA
>>> import utool as ut
>>> modname = 'lru'
>>> pipiname = 'git+https://github.com/amitdev/lru-dict'
>>> lru = ut.tryimport(modname, pipiname, ensure=True)
>>> assert isinstance(lru, types.ModuleType), 'did not ensure lru'

utool.util_inject module

Injects code into live modules or into text source files.

Basic use case is to extend the print function into a logging function

utool.util_inject.DUMMYPROF_FUNC(func)[source]

dummy profiling func. does nothing

utool.util_inject.KERNPROF_FUNC(func)

dummy profiling func. does nothing

utool.util_inject.TIMERPROF_FUNC(func)[source]
utool.util_inject.colored_pygments_excepthook(type_, value, tb)[source]

References

https://stackoverflow.com/questions/14775916/color-exceptions-python

CommandLine:

python -m utool.util_inject --test-colored_pygments_excepthook
utool.util_inject.get_injected_modules()[source]
utool.util_inject.inject(module_name=None, module_prefix=u'[???]', DEBUG=False, module=None, N=1)[source]

Injects your module with utool magic

Utool magic is not actually magic. It just turns your print statments into logging statments, allows for your module to be used with the utool.Indent context manager and the and utool.indent_func decorator. printDBG will soon be deprecated as will print_. The function rrr is a developer convinience for reloading the module dynamically durring runtime. The profile decorator is a no-op if not using kernprof.py, otherwise it is kernprof.py’s profile decorator.

Parameters:
  • module_name (str) – the __name__ varaible in your module
  • module_prefix (str) – a user defined module prefix
  • DEBUG (bool) –
  • module (None) – the actual module (optional)
Returns:

(print, print_, printDBG, rrr, profile_)
Return type:

tuple

Example

>>> from utool.util_inject import *  # NOQA
>>> from __future__ import absolute_import, division, print_function, unicode_literals
>>> from util.util_inject import inject
>>> print, rrr, profile = inject2(__name__, '[mod]')
utool.util_inject.inject2(module_name=None, module_prefix=u'[???]', DEBUG=False, module=None, N=1)[source]

wrapper that depricates print_ and printDBG

utool.util_inject.inject_all(DEBUG=False)[source]

UNFINISHED. DO NOT USE

Injects the print, print_, printDBG, rrr, and profile functions into all loaded modules

utool.util_inject.inject_colored_exceptions()[source]

Causes exceptions to be colored if not already

Hooks into sys.excepthook

CommandLine:

python -m utool.util_inject --test-inject_colored_exceptions

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inject import *  # NOQA
>>> print('sys.excepthook = %r ' % (sys.excepthook,))
>>> #assert sys.excepthook is colored_pygments_excepthook, 'bad excepthook'
>>> raise Exception('should be in color')
utool.util_inject.inject_print_func2(module)[source]
utool.util_inject.inject_print_functions(module_name=None, module_prefix=u'[???]', DEBUG=False, module=None)[source]

makes print functions to be injected into the module

utool.util_inject.inject_profile_function(module_name=None, module_prefix=u'[???]', module=None)[source]
utool.util_inject.inject_python_code(fpath, patch_code, tag=None, inject_location=u'after_imports')[source]

DEPRICATE puts code into files on disk

utool.util_inject.inject_python_code2(fpath, patch_code, tag)[source]

Does autogeneration stuff

utool.util_inject.inject_reload_function(module_name=None, module_prefix=u'[???]', module=None)[source]

Injects dynamic module reloading

utool.util_inject.memprof(func)[source]

requires memory_profiler pip install memory_profiler

References

https://pypi.python.org/pypi/memory_profiler

utool.util_inject.noinject(module_name=None, module_prefix=u'[???]', DEBUG=False, module=None, N=0, via=None)[source]

Use in modules that do not have inject in them

Does not inject anything into the module. Just lets utool know that a module is being imported so the import order can be debuged

utool.util_inject.print(*args)

logging builtins.print

utool.util_inject.profile(func)
utool.util_inject.rrr(verbose=True)

Dynamic module reloading

utool.util_inject.split_python_text_into_lines(text)[source]

# TODO: make it so this function returns text so one statment is on one # line that means no splitting up things like function definitions into # multiple lines

utool.util_inspect module

class utool.util_inspect.KWReg(kwreg, enabled=False)[source]

Bases: object

Helper to register keywords for complex keyword parsers

defaultkw
print_defaultkw(kwreg)[source]
utool.util_inspect.check_module_usage(modpath_partterns)[source]
Parameters:modpath_partterns (list) –

CommandLine:

python -m utool.util_inspect --exec-check_module_usage --show
utprof.py -m utool.util_inspect --exec-check_module_usage --show
python -m utool.util_inspect --exec-check_module_usage --pat="['auto*', 'user_dialogs.py', 'special_query.py', 'qt_inc_automatch.py', 'devcases.py']"
python -m utool.util_inspect --exec-check_module_usage --pat="preproc_detectimg.py"
python -m utool.util_inspect --exec-check_module_usage --pat="neighbor_index.py"

Ignore:

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> modpath_partterns = ['_grave*']
>>> modpath_partterns = ['auto*', 'user_dialogs.py', 'special_query.py', 'qt_inc_automatch.py', 'devcases.py']
>>> modpath_partterns = ut.get_argval('--pat', type_=list, default=['*'])
>>> modpath_partterns = ['neighbor_index.py']
>>> result = check_module_usage(modpath_partterns)
>>> print(result)
utool.util_inspect.dummy_func(arg1, arg2, arg3=None, arg4=[1, 2, 3], arg5={}, **kwargs)[source]

test func for kwargs parseing

utool.util_inspect.exec_func_src(func, globals_=None, locals_=None, key_list=None, sentinal=None, update=None)[source]

execs a func and returns requested local vars.

Does not modify globals unless update=True (or in IPython)

SeeAlso:
ut.execstr_funckw
utool.util_inspect.filter_valid_kwargs(func, dict_)[source]
utool.util_inspect.find_child_kwarg_funcs(sourcecode, target_kwargs_name=u'kwargs')[source]

CommandLine:

python3 -m utool.util_inspect --exec-find_child_kwarg_funcs

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> sourcecode = ut.codeblock(
    '''
    warped_patch1_list, warped_patch2_list = list(zip(*ut.ichunks(data, 2)))
    interact_patches(labels, warped_patch1_list, warped_patch2_list, flat_metadata, **kwargs)
    import sys
    sys.badcall(**kwargs)
    def foo():
        bar(**kwargs)
        ut.holymoly(**kwargs)
        baz()
        def biz(**kwargs):
            foo2(**kwargs)
    ''')
>>> child_funcnamess = ut.find_child_kwarg_funcs(sourcecode)
>>> print('child_funcnamess = %r' % (child_funcnamess,))
>>> assert 'foo2' not in child_funcnamess, 'foo2 should not be found'
>>> assert 'bar' in child_funcnamess, 'bar should be found'

Notes:

utool.util_inspect.find_pattern_above_row(pattern, line_list, row, maxIter=50)[source]

originally part of the vim plugin

utool.util_inspect.find_pyclass_above_row(line_list, row)[source]

originally part of the vim plugin

utool.util_inspect.find_pyfunc_above_row(line_list, row, orclass=False)[source]

originally part of the vim plugin

CommandLine:

python -m utool.util_inspect --test-find_pyfunc_above_row

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> func = find_pyfunc_above_row
>>> fpath = meta_util_six.get_funcglobals(func)['__file__'].replace('.pyc', '.py')
>>> line_list = ut.read_from(fpath, aslines=True)
>>> row = meta_util_six.get_funccode(func).co_firstlineno + 1
>>> pyfunc, searchline = find_pyfunc_above_row(line_list, row)
>>> result = pyfunc
>>> print(result)
find_pyfunc_above_row

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> fpath = ut.util_inspect.__file__.replace('.pyc', '.py')
>>> line_list = ut.read_from(fpath, aslines=True)
>>> row = 1608
>>> pyfunc, searchline = find_pyfunc_above_row(line_list, row, orclass=True)
>>> result = pyfunc
>>> print(result)
find_pyfunc_above_row
utool.util_inspect.get_argnames(func)[source]
utool.util_inspect.get_dev_hints()[source]
utool.util_inspect.get_docstr(func_or_class)[source]

Get the docstring from a live object

utool.util_inspect.get_func_argspec(func)[source]

wrapper around inspect.getargspec but takes into account utool decorators

utool.util_inspect.get_func_kwargs(func, stripdef=False, stripret=False, strip_docstr=False, remove_linenums=None)[source]

func = ibeis.run_experiment

utool.util_inspect.get_func_modname(func)[source]
utool.util_inspect.get_func_sourcecode(func, stripdef=False, stripret=False, strip_docstr=False, strip_comments=False, remove_linenums=None)[source]

wrapper around inspect.getsource but takes into account utool decorators strip flags are very hacky as of now

Parameters:
  • func (function) –
  • stripdef (bool) –
  • stripret (bool) – (default = False)
  • strip_docstr (bool) – (default = False)
  • strip_comments (bool) – (default = False)
  • remove_linenums (None) – (default = None)

CommandLine:

python -m utool.util_inspect --test-get_func_sourcecode

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> # build test data
>>> func = get_func_sourcecode
>>> stripdef = True
>>> stripret = True
>>> sourcecode = get_func_sourcecode(func, stripdef)
>>> # verify results
>>> print(result)
utool.util_inspect.get_funcdoc(func)[source]
utool.util_inspect.get_funcfpath(func)[source]
utool.util_inspect.get_funcglobals(func)[source]
utool.util_inspect.get_funcname(func)[source]
utool.util_inspect.get_imfunc(func)[source]
utool.util_inspect.get_kwargs(func)[source]
Parameters:func (function) –
Returns:keys, is_arbitrary keys (list): kwargs keys is_arbitrary (bool): has generic **kwargs
Return type:tuple

CommandLine:

python -m utool.util_inspect --test-get_kwargs

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> # build test data
>>> func = '?'
>>> result = get_kwargs(func)
>>> # verify results
>>> print(result)
utool.util_inspect.get_kwdefaults(func, parse_source=False)[source]
Parameters:func (func) –
Returns:
Return type:dict

CommandLine:

python -m utool.util_inspect --exec-get_kwdefaults

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> func = dummy_func
>>> parse_source = True
>>> kwdefaults = get_kwdefaults(func, parse_source)
>>> print('kwdefaults = %s' % (ut.dict_str(kwdefaults),))
utool.util_inspect.get_kwdefaults2(func, parse_source=False)[source]
utool.util_inspect.get_module_owned_functions(module)[source]

Replace with iter_module_doctesable (but change that name to be something better)

returns functions actually owned by the module module = vtool.distance

utool.util_inspect.get_unbound_args(argspec)[source]
utool.util_inspect.help_members(obj)[source]
Parameters:obj (class or module) –

CommandLine:

python -m utool.util_inspect --exec-help_members

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> obj = ut.DynStruct
>>> result = help_members(obj)
>>> print(result)
utool.util_inspect.infer_arg_types_and_descriptions(argname_list, defaults)[source]
Parameters:
  • argname_list (list) –
  • defaults (list) –
Returns:

(argtype_list, argdesc_list)
Return type:

tuple

CommandLine:

python -m utool.util_inspect --test-infer_arg_types_and_descriptions

Example

>>> # ENABLE_DOCTEST
>>> import utool
>>> argname_list = ['ibs', 'qaid', 'fdKfds', 'qfx2_foo']
>>> defaults = None
>>> tup = utool.infer_arg_types_and_descriptions(argname_list, defaults)
>>> argtype_list, argdesc_list, argdefault_list, hasdefault_list = tup
utool.util_inspect.infer_function_info(func)[source]

Infers information for make_default_docstr # TODO: Interleave old documentation with new documentation

Parameters:func (function) – live python function

CommandLine:

python -m utool --tf infer_function_info:0
python -m utool --tf infer_function_info:1 --funcname=ibeis_cnn.models.siam.ignore_hardest_cases
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> func = ut.infer_function_info
>>> #func = ut.Timer.tic
>>> func = ut.make_default_docstr
>>> funcinfo = infer_function_info(func)
>>> result = ut.dict_str(funcinfo.__dict__)
>>> print(result)
Example1:
>>> # SCRIPT
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> funcname = ut.get_argval('--funcname')
>>> # Parse out custom function
>>> modname = '.'.join(funcname.split('.')[0:-1])
>>> script = 'import {modname}\nfunc = {funcname}'.format(
>>>     modname=modname, funcname=funcname)
>>> globals_, locals_ = {}, {}
>>> exec(script, globals_, locals_)
>>> func = locals_['func']
>>> funcinfo = infer_function_info(func)
>>> result = ut.dict_str(funcinfo.__dict__)
>>> print(result)
utool.util_inspect.inherit_kwargs(inherit_func)[source]

TODO move to util_decor inherit_func = inspect_pdfs func = encoder.visualize.im_func

utool.util_inspect.is_bateries_included(item)[source]

Returns if a value is a python builtin function

Parameters:item (object) –
Returns:flag
Return type:bool

References

http://stackoverflow.com/questions/23149218/check-if-a-python-function-is-builtin

CommandLine:

python -m utool._internal.meta_util_six --exec-is_builtin

Example

>>> # DISABLE_DOCTEST
>>> from utool._internal.meta_util_six import *  # NOQA
>>> item = zip
>>> flag = is_bateries_included(item)
>>> result = ('flag = %s' % (str(flag),))
>>> print(result)
utool.util_inspect.is_defined_by_module(item, module)[source]

Check if item is directly defined by a module. This check may be prone to errors.

utool.util_inspect.is_valid_python(code, reraise=True, ipy_magic_workaround=False)[source]

References

http://stackoverflow.com/questions/23576681/python-check-syntax

utool.util_inspect.iter_module_doctestable(module, include_funcs=True, include_classes=True, include_methods=True, include_builtin=True, include_inherited=False, debug_key=None)[source]

Yeilds doctestable live object form a modules

TODO: change name to iter_module_members Replace with iter_module_doctesable (but change that name to be something better)

Args:
module (module): live python module include_funcs (bool): include_classes (bool): include_methods (bool): include_builtin (bool): (default = True) include_inherited (bool): (default = False)
Yeilds:
tuple (str, callable): (funcname, func) doctestable
CommandLine:
python -m utool –tf iter_module_doctestable
–modname=ibeis.algo.hots.chip_match –modname=ibeis.control.IBEISControl –modname=ibeis.control.SQLDatabaseControl –modname=ibeis.control.manual_annot_funcs –modname=ibeis.control.manual_annot_funcs –modname=ibeis.expt.test_result –modname=utool.util_progress –debug-key=build_msg_fmtstr_time2 –modname=utool.util_progress –debug-key=ProgressIter
Debug:
# fix profile with doctest utprof.py -m utool –tf iter_module_doctestable –modname=utool.util_inspect –debugkey=zzz_profiled_is_yes
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *   # NOQA
>>> import utool as ut
>>> modname = ut.get_argval('--modname', type_=str, default=None)
>>> kwargs = ut.argparse_funckw(iter_module_doctestable)
>>> module = ut.util_tests if modname is None else ut.import_modname(modname)
>>> debug_key = ut.get_argval('--debugkey', type_=str, default=None)
>>> kwargs['debug_key'] = debug_key
>>> doctestable_list = list(iter_module_doctestable(module, **kwargs))
>>> func_names = sorted(ut.get_list_column(doctestable_list, 0))
>>> print(ut.list_str(func_names))
utool.util_inspect.list_class_funcnames(fname, blank_pats=[u' #'])[source]
Parameters:
  • fname (str) – filepath
  • blank_pats (list) – defaults to ‘ #’
Returns:

funcname_list
Return type:

list

Example

>>> from utool.util_inspect import *  # NOQA
>>> fname = 'util_class.py'
>>> blank_pats = ['    #']
>>> funcname_list = list_class_funcnames(fname, blank_pats)
>>> print(funcname_list)
utool.util_inspect.list_global_funcnames(fname, blank_pats=[u' #'])[source]
Parameters:
  • fname (str) – filepath
  • blank_pats (list) – defaults to ‘ #’
Returns:

funcname_list
Return type:

list

Example

>>> from utool.util_inspect import *  # NOQA
>>> fname = 'util_class.py'
>>> blank_pats = ['    #']
>>> funcname_list = list_global_funcnames(fname, blank_pats)
>>> print(funcname_list)
utool.util_inspect.lookup_attribute_chain(attrname, namespace)[source]
>>> attrname = funcname
>>> namespace = mod.__dict__

>>> import utool as ut
>>> globals_ = ut.util_inspect.__dict__
>>> attrname = 'KWReg.print_defaultkw'
utool.util_inspect.parse_callname(searchline, sentinal=u'def ')[source]

Parses the function or class name from a signature line originally part of the vim plugin

utool.util_inspect.parse_func_kwarg_keys(func, with_vals=False)[source]

hacky inference of kwargs keys

utool.util_inspect.parse_kwarg_keys(source, keywords=u'kwargs', with_vals=False)[source]

very hacky way to infer some of the kwarg keys

TODO: use a code parse tree here. Use hints. Find other docstrings of functions that are called with kwargs. Find the name of the kwargs variable.

Parameters:source (str) –
Returns:kwarg_keys
Return type:list

CommandLine:

python -m utool.util_inspect --exec-parse_kwarg_keys

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> source = "\n  kwargs.get('foo', None)\n  kwargs.pop('bar', 3)\n kwargs.pop('str', '3fd')\n kwargs.pop('str', '3f\'d')\n  \"kwargs.get('baz', None)\""
>>> print(source)
>>> kwarg_keys = parse_kwarg_keys(source, with_vals=True)
>>> result = ('kwarg_keys = %s' % (str(kwarg_keys),))
>>> print(result)
utool.util_inspect.parse_return_type(sourcecode)[source]
Parameters:sourcecode –
Returns:(return_type, return_name, return_header)
Return type:tuple

CommandLine:

python -m utool.util_inspect --exec-parse_return_type
python -m utool.util_inspect --test-parse_return_type

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> sourcecode = ut.codeblock(
... 'def foo(tmp=False):\n'
... '    bar = True\n'
... '    return bar\n'
... )
>>> returninfo = parse_return_type(sourcecode)
>>> result = ut.repr2(returninfo)
>>> print(result)
('?', 'bar', 'Returns', '')

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> sourcecode = ut.codeblock(
... 'def foo(tmp=False):\n'
... '    return True\n'
... )
>>> returninfo = parse_return_type(sourcecode)
>>> result = ut.repr2(returninfo)
>>> print(result)
('bool', 'True', 'Returns', '')

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> sourcecode = ut.codeblock(
... 'def foo(tmp=False):\n'
... '    for i in range(2): \n'
... '        yield i\n'
... )
>>> returninfo = parse_return_type(sourcecode)
>>> result = ut.repr2(returninfo)
>>> print(result)
('?', 'i', 'Yields', '')

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> sourcecode = ut.codeblock(
... 'def foo(tmp=False):\n'
... '    if tmp is True:\n'
... '        return (True, False)\n'
... '    elif tmp is False:\n'
... '        return 1\n'
... '    else:\n'
... '        bar = baz()\n'
... '        return bar\n'
... )
>>> returninfo = parse_return_type(sourcecode)
>>> result = ut.repr2(returninfo)
>>> print(result)
('tuple', '(True, False)', 'Returns', '')
utool.util_inspect.prettyprint_parsetree(pt)[source]

pip install astdump pip install codegen

utool.util_inspect.recursive_parse_kwargs(root_func, path_=None)[source]

recursive kwargs parser FIXME: if docstr indentation is off, this fails

Parameters:
  • root_func (function) – live python function
  • path (None) – (default = None)
Returns:
Return type:

list

CommandLine:

python -m utool.util_inspect --exec-recursive_parse_kwargs:0
python -m utool.util_inspect --exec-recursive_parse_kwargs:1
python -m utool.util_inspect --exec-recursive_parse_kwargs:2 --mod plottool --func draw_histogram

python -m utool.util_inspect --exec-recursive_parse_kwargs:2 --mod vtool --func ScoreNormalizer.visualize

python -m utool.util_inspect --exec-recursive_parse_kwargs:2 --mod ibeis.viz.viz_matches --func show_name_matches --verbinspect

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> root_func = iter_module_doctestable
>>> path_ = None
>>> result = ut.repr2(recursive_parse_kwargs(root_func), nl=1)
>>> print(result)
[
    ('include_funcs', True),
    ('include_classes', True),
    ('include_methods', True),
    ('include_builtin', True),
    ('include_inherited', False),
    ('debug_key', None),
]

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> from ibeis.algo.hots import chip_match
>>> import utool as ut
>>> root_func = chip_match.ChipMatch.show_ranked_matches
>>> path_ = None
>>> result = ut.repr2(recursive_parse_kwargs(root_func))
>>> print(result)

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_inspect import *  # NOQA
>>> import utool as ut
>>> modname = ut.get_argval('--mod', type_=str, default='plottool')
>>> funcname = ut.get_argval('--func', type_=str, default='draw_histogram')
>>> mod = ut.import_modname(modname)
>>> root_func = lookup_attribute_chain(funcname, mod.__dict__)
>>> path_ = None
>>> parsed = recursive_parse_kwargs(root_func)
>>> result = ut.repr2(parsed)
>>> print(result)
utool.util_inspect.set_funcdoc(func, newdoc)[source]
utool.util_inspect.set_funcname(func, newname)[source]
utool.util_inspect.zzz_profiled_is_no()[source]
utool.util_inspect.zzz_profiled_is_yes()[source]

utool.util_io module

utool.util_io.load_cPkl(fpath, verbose=None, n=2)[source]

Loads a pickled file with optional verbosity

utool.util_io.load_data(fpath, **kwargs)[source]

More generic interface to load data

utool.util_io.load_hdf5(fpath, verbose=False)[source]
utool.util_io.load_numpy(fpath, mmap_mode=None, verbose=None)[source]
utool.util_io.load_pytables(fpath, verbose=False)[source]
utool.util_io.load_text(fpath, verbose=None, aslines=False, strict=True, n=None, errors=u'replace')

Reads text from a file. Automatically returns utf8.

Parameters:
  • fpath (str) – file path
  • aslines (bool) – if True returns list of lines
  • verbose (bool) – verbosity flag
Returns:

text from fpath (this is unicode)
Return type:

str
utool.util_io.lock_and_load_cPkl(fpath, verbose=False)[source]
utool.util_io.lock_and_save_cPkl(fpath, data, verbose=False)[source]
utool.util_io.read_from(fpath, verbose=None, aslines=False, strict=True, n=None, errors=u'replace')[source]

Reads text from a file. Automatically returns utf8.

Parameters:
  • fpath (str) – file path
  • aslines (bool) – if True returns list of lines
  • verbose (bool) – verbosity flag
Returns:

text from fpath (this is unicode)
Return type:

str
utool.util_io.read_lines_from(fpath, num_lines=None, verbose=None, n=3)[source]
utool.util_io.readfrom(fpath, verbose=None, aslines=False, strict=True, n=None, errors=u'replace')

Reads text from a file. Automatically returns utf8.

Parameters:
  • fpath (str) – file path
  • aslines (bool) – if True returns list of lines
  • verbose (bool) – verbosity flag
Returns:

text from fpath (this is unicode)
Return type:

str
utool.util_io.save_cPkl(fpath, data, verbose=None, n=2)[source]

Saves data to a pickled file with optional verbosity

utool.util_io.save_data(fpath, data, **kwargs)[source]

More generic interface to write data

utool.util_io.save_hdf5(fpath, data, verbose=False, compression=u'lzf')[source]

Restricted save of data using hdf5. Can only save ndarrays and dicts of ndarrays.

Parameters:
  • fpath (str) –
  • data (ndarray) –
  • compression (str) – DEFLATE/GZIP - standard LZF - fast SHUFFLE - compression ratio FLETCHER32 - error detection Scale-offset - integer / float scaling and truncation SZIP - fast and patented

CommandLine:

python -m utool.util_io --test-save_hdf5

References

http://docs.h5py.org/en/latest/quick.html http://docs.h5py.org/en/latest/mpi.html

Example

>>> # ENABLE_IF HAS_H5PY
>>> from utool.util_io import *  # NOQA
>>> import numpy as np
>>> import utool as ut
>>> # build test data
>>> rng = np.random.RandomState(0)
>>> data = (rng.rand(100000, 128) * 255).astype(np.uint8).copy()
>>> verbose = True
>>> fpath = ut.unixjoin(ut.ensure_app_resource_dir('utool'), 'myfile.hdf5')
>>> compression = 'lzf'
>>> # execute function
>>> ut.delete(fpath)
>>> save_hdf5(fpath, data, verbose, compression)
>>> data2 = load_hdf5(fpath, verbose)
>>> assert data is not data2
>>> assert np.all(data == data2)
>>> assert ut.delete(fpath)
Timeit:
>>> # cPkl / numpy seems to be faster with this initial implementation
>>> import utool as ut
>>> data = (rng.rand(1000000, 128) * 255).astype(np.uint8).copy()
>>> print(ut.get_object_size_str(data))
>>> del data
>>> setup = ut.codeblock(
>>>     '''
        import numpy as np
        import utool as ut
        rng = np.random.RandomState(0)
        fpath = ut.unixjoin(ut.ensure_app_resource_dir('utool'), 'io_test_data')
        data = (rng.rand(1000000, 128) * 255).astype(np.uint8).copy()
        #print(ut.get_object_size_str(data))
        ''')
>>> # Test load time
>>> stmt_list1 = ut.codeblock(
>>>     '''
        ut.save_hdf5(fpath + '.hdf5', data, verbose=False, compression='gzip')
        ut.save_hdf5(fpath + '.hdf5', data, verbose=False, compression='lzf')
        ut.save_cPkl(fpath + '.cPkl', data, verbose=False)
        ut.save_numpy(fpath + '.npy', data, verbose=False)
        ut.save_pytables(fpath + '.tables', data, verbose=False)
        ''').split('\n')
>>> ut.util_dev.timeit_compare(stmt_list1, setup, int(10))
>>> # Test save time
>>> stmt_list2 = ut.codeblock(
>>>     '''
        ut.load_hdf5(fpath + '.hdf5', verbose=False)
        ut.load_cPkl(fpath + '.cPkl', verbose=False)
        ut.load_numpy(fpath + '.npy', verbose=False)
        ut.load_pytables(fpath + '.tables', verbose=False)
        ''').split('\n')
>>> ut.util_dev.timeit_compare(stmt_list2, setup, int(10))
>>> print('finished timeing')
+----------------
| TIMEIT COMPARE
+----------------
| iterations = 10
| Input:
|     | num | stmt
|     |   0 | u"ut.save_hdf5(fpath + '.hdf5', data, verbose=False, compression='gzip')"
|     |   1 | u"ut.save_hdf5(fpath + '.hdf5', data, verbose=False, compression='lzf')"
|     |   2 | u"ut.save_cPkl(fpath + '.cPkl', data, verbose=False)"
|     |   3 | u"ut.save_numpy(fpath + '.npy', data, verbose=False)"
|     |   4 | u"ut.save_pytables(fpath + '.tables', data, verbose=False)"
...
| Output:
|    * PASSED: each statement produced the same result
|    | num | total time | per loop | stmt
|    |   0 |    0.03 ks |   3.15 s | ut.save_hdf5(fpath + '.hdf5', data, verbose=False, compression='gzip')
|    |   1 |    0.01 ks |   1.25 s | ut.save_hdf5(fpath + '.hdf5', data, verbose=False, compression='lzf')
|    |   2 |     5.30 s |   0.53 s | ut.save_cPkl(fpath + '.cPkl', data, verbose=False)
|    |   3 |     4.97 s |   0.50 s | ut.save_numpy(fpath + '.npy', data, verbose=False)
|    |   4 |     9.23 s |   0.92 s | ut.save_pytables(fpath + '.tables', data, verbose=False)
L_________________
+----------------
| TIMEIT COMPARE
+----------------
| iterations = 10
| Input:
|     | num | stmt
|     |   0 | u"ut.load_hdf5(fpath + '.hdf5', verbose=False)"
|     |   1 | u"ut.load_cPkl(fpath + '.cPkl', verbose=False)"
|     |   2 | u"ut.load_numpy(fpath + '.npy', verbose=False)"
|     |   3 | u"ut.load_pytables(fpath + '.tables', verbose=False)"
...
| Output:
|    * PASSED: each statement produced the same result
|    | num | total time | per loop | stmt
|    |   0 |     2.39 s |   0.24 s | ut.load_hdf5(fpath + '.hdf5', verbose=False)
|    |   1 |     0.39 s |   0.04 s | ut.load_cPkl(fpath + '.cPkl', verbose=False)
|    |   2 |     0.19 s |   0.02 s | ut.load_numpy(fpath + '.npy', verbose=False)
|    |   3 |     0.33 s |   0.03 s | ut.load_pytables(fpath + '.tables', verbose=False)
L_________________

Notes

pip install mpi4py

utool.util_io.save_numpy(fpath, data, verbose=None)[source]
utool.util_io.save_pytables(fpath, data, verbose=False)[source]

sudo pip install numexpr sudo pip install tables

References

https://pytables.github.io/cookbook/py2exe_howto.html https://gist.github.com/andrewgiessel/7515520 http://stackoverflow.com/questions/8843062/python-how-to-store-a-numpy-multidimensional-array-in-pytables http://pytables.github.io/usersguide/tutorials.html#creating-new-array-objects

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_io import *  # NOQA
>>> import numpy as np
>>> import utool as ut
>>> # build test data
>>> verbose = True
>>> fpath = 'myfile.pytables.hdf5'
>>> np.random.seed(0)
>>> compression = 'gzip'
>>> data = (np.random.rand(100000, 128) * 255).astype(np.uint8).copy()
>>> # execute function
>>> ut.delete(fpath)
>>> save_pytables(fpath, data, verbose)
>>> data2 = load_pytables(fpath, verbose)
>>> assert data is not data2
>>> assert np.all(data == data2)
>>> assert ut.delete(fpath)
utool.util_io.save_text(fpath, to_write, aslines=False, verbose=None, onlyifdiff=False, mode=u'w', n=2)

Writes text to a file. Automatically encodes text as utf8.

Parameters:
  • fpath (str) – file path
  • to_write (str) – text to write (must be unicode text)
  • aslines (bool) – if True to_write is assumed to be a list of lines
  • verbose (bool) – verbosity flag
  • onlyifdiff (bool) – only writes if needed! checks hash of to_write vs the hash of the contents of fpath
  • mode (unicode) – (default = u’w’)
  • n (int) – (default = 2)

CommandLine:

python -m utool.util_io --exec-write_to --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_io import *  # NOQA
>>> import utool as ut
>>> fpath = ut.unixjoin(ut.get_app_resource_dir('utool'), 'testwrite.txt')
>>> ut.delete(fpath)
>>> to_write = 'utf-8 symbols Δ, Й, ק, م, ๗, あ, 叶, 葉, and 말.'
>>> aslines = False
>>> verbose = True
>>> onlyifdiff = False
>>> mode = u'w'
>>> n = 2
>>> write_to(fpath, to_write, aslines, verbose, onlyifdiff, mode, n)
>>> read_ = ut.read_from(fpath)
>>> print('read_    = ' + read_)
>>> print('to_write = ' + to_write)
>>> assert read_ == to_write
utool.util_io.try_decode(x)[source]
utool.util_io.write_to(fpath, to_write, aslines=False, verbose=None, onlyifdiff=False, mode=u'w', n=2)[source]

Writes text to a file. Automatically encodes text as utf8.

Parameters:
  • fpath (str) – file path
  • to_write (str) – text to write (must be unicode text)
  • aslines (bool) – if True to_write is assumed to be a list of lines
  • verbose (bool) – verbosity flag
  • onlyifdiff (bool) – only writes if needed! checks hash of to_write vs the hash of the contents of fpath
  • mode (unicode) – (default = u’w’)
  • n (int) – (default = 2)

CommandLine:

python -m utool.util_io --exec-write_to --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_io import *  # NOQA
>>> import utool as ut
>>> fpath = ut.unixjoin(ut.get_app_resource_dir('utool'), 'testwrite.txt')
>>> ut.delete(fpath)
>>> to_write = 'utf-8 symbols Δ, Й, ק, م, ๗, あ, 叶, 葉, and 말.'
>>> aslines = False
>>> verbose = True
>>> onlyifdiff = False
>>> mode = u'w'
>>> n = 2
>>> write_to(fpath, to_write, aslines, verbose, onlyifdiff, mode, n)
>>> read_ = ut.read_from(fpath)
>>> print('read_    = ' + read_)
>>> print('to_write = ' + to_write)
>>> assert read_ == to_write
utool.util_io.writeto(fpath, to_write, aslines=False, verbose=None, onlyifdiff=False, mode=u'w', n=2)

Writes text to a file. Automatically encodes text as utf8.

Parameters:
  • fpath (str) – file path
  • to_write (str) – text to write (must be unicode text)
  • aslines (bool) – if True to_write is assumed to be a list of lines
  • verbose (bool) – verbosity flag
  • onlyifdiff (bool) – only writes if needed! checks hash of to_write vs the hash of the contents of fpath
  • mode (unicode) – (default = u’w’)
  • n (int) – (default = 2)

CommandLine:

python -m utool.util_io --exec-write_to --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_io import *  # NOQA
>>> import utool as ut
>>> fpath = ut.unixjoin(ut.get_app_resource_dir('utool'), 'testwrite.txt')
>>> ut.delete(fpath)
>>> to_write = 'utf-8 symbols Δ, Й, ק, م, ๗, あ, 叶, 葉, and 말.'
>>> aslines = False
>>> verbose = True
>>> onlyifdiff = False
>>> mode = u'w'
>>> n = 2
>>> write_to(fpath, to_write, aslines, verbose, onlyifdiff, mode, n)
>>> read_ = ut.read_from(fpath)
>>> print('read_    = ' + read_)
>>> print('to_write = ' + to_write)
>>> assert read_ == to_write

utool.util_ipynb module

Utils for IPython/Jupyter Notebooks

utool.util_ipynb.code_cell(sourcecode)[source]
Parameters:sourcecode (str) –
Returns:json formatted ipython notebook code cell
Return type:str

CommandLine:

python -m ibeis.templates.generate_notebook --exec-code_cell

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.templates.generate_notebook import *  # NOQA
>>> sourcecode = notebook_cells.timestamp_distribution[1]
>>> sourcecode = notebook_cells.initialize[1]
>>> result = code_cell(sourcecode)
>>> print(result)
utool.util_ipynb.export_notebook(run_nb, fname)[source]
utool.util_ipynb.format_cells(block, locals_=None)[source]
utool.util_ipynb.make_autogen_str()[source]
Returns:
Return type:str

CommandLine:

python -m utool.util_ipynb --exec-make_autogen_str --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_ipynb import *  # NOQA
>>> import utool as ut
>>> result = make_autogen_str()
>>> print(result)
utool.util_ipynb.make_notebook(cell_list)[source]

References

# Change cell width http://stackoverflow.com/questions/21971449/how-do-i-increase-the-cell-width-of-the-ipython-notebook-in-my-browser/24207353#24207353

utool.util_ipynb.markdown_cell(markdown)[source]
Parameters:markdown (str) –
Returns:json formatted ipython notebook markdown cell
Return type:str

CommandLine:

python -m ibeis.templates.generate_notebook --exec-markdown_cell

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.templates.generate_notebook import *  # NOQA
>>> markdown = '# Title'
>>> result = markdown_cell(markdown)
>>> print(result)
utool.util_ipynb.repr_single_for_md(s)[source]
Parameters:s (str) –
Returns:str_repr
Return type:str

CommandLine:

python -m ibeis.templates.generate_notebook --exec-repr_single_for_md --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.templates.generate_notebook import *  # NOQA
>>> s = '#HTML(\'<iframe src="%s" width=700 height=350></iframe>\' % pdf_fpath)'
>>> result = repr_single_for_md(s)
>>> print(result)
utool.util_ipynb.run_ipython_notebook(notebook_str)[source]

References

https://github.com/paulgb/runipy >>> from utool.util_ipynb import * # NOQA

utool.util_iter module

utool.util_iter.and_iters(*args)[source]
utool.util_iter.evaluate_generator(iter_)[source]

for evaluating each item in a generator and ignoring output

utool.util_iter.ichunks(iterable, chunksize, bordermode=None)[source]

generates successive n-sized chunks from iterable.

Parameters:
  • iterable (list) – input to iterate over
  • chunksize (int) – size of sublist to return
  • bordermode (str) – None, ‘cycle’, or ‘replicate’

References

http://stackoverflow.com/questions/434287/iterate-over-a-list-in-chunks

SeeAlso:
util_progress.get_nTotalChunks

CommandLine:

python -m utool.util_iter --exec-ichunks --show
Timeit:
>>> import utool as ut
>>> setup = ut.codeblock('''
        from utool.util_iter import *  # NOQA
        iterable = list(range(100))
        chunksize = 8
        ''')
>>> stmt_list = [
...     'list(ichunks(iterable, chunksize))',
...     'list(ichunks_noborder(iterable, chunksize))',
...     'list(ichunks_list(iterable, chunksize))',
... ]
>>> (passed, times, results) = ut.timeit_compare(stmt_list, setup)

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> iterable = [1, 2, 3, 4, 5, 6, 7]
>>> chunksize = 3
>>> genresult = ichunks(iterable, chunksize)
>>> result = list(genresult)
>>> print(result)
[[1, 2, 3], [4, 5, 6], [7]]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> iterable = (1, 2, 3, 4, 5, 6, 7)
>>> chunksize = 3
>>> bordermode = 'cycle'
>>> genresult = ichunks(iterable, chunksize, bordermode)
>>> result = list(genresult)
>>> print(result)
[[1, 2, 3], [4, 5, 6], [7, 1, 2]]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> iterable = (1, 2, 3, 4, 5, 6, 7)
>>> chunksize = 3
>>> bordermode = 'replicate'
>>> genresult = ichunks(iterable, chunksize, bordermode)
>>> result = list(genresult)
>>> print(result)
[[1, 2, 3], [4, 5, 6], [7, 7, 7]]
utool.util_iter.ichunks_cycle(iterable, chunksize)[source]
utool.util_iter.ichunks_list(list_, chunksize)[source]

input must be a list.

SeeAlso:
ichunks

References

http://stackoverflow.com/questions/434287/iterate-over-a-list-in-chunks

utool.util_iter.ichunks_noborder(iterable, chunksize)[source]
utool.util_iter.ichunks_replicate(iterable, chunksize)[source]
utool.util_iter.ifilter_Nones(iter_)[source]

Removes any nones from the iterable

utool.util_iter.ifilter_items(item_iter, flag_iter)

iter_compress - like numpy compress

Parameters:
  • item_iter (list) –
  • flag_iter (list) – of bools
Returns:

true_items
Return type:

list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> item_iter = [1, 2, 3, 4, 5]
>>> flag_iter = [False, True, True, False, True]
>>> true_items = iter_compress(item_iter, flag_iter)
>>> result = list(true_items)
>>> print(result)
[2, 3, 5]
utool.util_iter.ifilterfalse_items(item_iter, flag_iter)[source]
Parameters:
  • item_iter (list) –
  • flag_iter (list) – of bools

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> item_iter = [1, 2, 3, 4, 5]
>>> flag_iter = [False, True, True, False, True]
>>> false_items = ifilterfalse_items(item_iter, flag_iter)
>>> result = list(false_items)
>>> print(result)
[1, 4]
utool.util_iter.iflatten(list_)[source]

flattens a list iteratively

utool.util_iter.iget_list_column(list_, colx)[source]

iterator version of get_list_column

utool.util_iter.iget_list_column_slice(list_, start=None, stop=None, stride=None)[source]

iterator version of get_list_column

utool.util_iter.interleave(args)[source]

zip followed by flatten

Parameters:args (tuple) – tuple of lists to interleave

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> import utool as ut
>>> args = ([1, 2, 3, 4, 5], ['A', 'B', 'C', 'D', 'E', 'F', 'G'])
>>> genresult = interleave(args)
>>> result = ut.list_str(list(genresult), nl=False)
>>> print(result)
[1, 'A', 2, 'B', 3, 'C', 4, 'D', 5, 'E']
utool.util_iter.iter_compress(item_iter, flag_iter)[source]

iter_compress - like numpy compress

Parameters:
  • item_iter (list) –
  • flag_iter (list) – of bools
Returns:

true_items
Return type:

list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> item_iter = [1, 2, 3, 4, 5]
>>> flag_iter = [False, True, True, False, True]
>>> true_items = iter_compress(item_iter, flag_iter)
>>> result = list(true_items)
>>> print(result)
[2, 3, 5]
utool.util_iter.iter_multichunks(iterable, chunksizes, bordermode=None)[source]

CommandLine:

python -m utool.util_iter --test-iter_multichunks

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> import utool as ut
>>> iterable = list(range(20))
>>> chunksizes = (3, 2, 3)
>>> bordermode = 'cycle'
>>> genresult = iter_multichunks(iterable, chunksizes, bordermode)
>>> multichunks = list(genresult)
>>> depthprofile = ut.depth_profile(multichunks)
>>> assert depthprofile[1:] == chunksizes, 'did not generate chunks correctly'
>>> result = ut.list_str(map(str, multichunks), nobr=True)
>>> print(result)
'[[[0, 1, 2], [3, 4, 5]], [[6, 7, 8], [9, 10, 11]], [[12, 13, 14], [15, 16, 17]]]',
'[[[18, 19, 0], [1, 2, 3]], [[4, 5, 6], [7, 8, 9]], [[10, 11, 12], [13, 14, 15]]]',
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> import utool as ut
>>> iterable = list(range(7))
>>> # when chunksizes is len == 1, then equlivalent to ichunks
>>> chunksizes = (3,)
>>> bordermode = 'cycle'
>>> genresult = iter_multichunks(iterable, chunksizes, bordermode)
>>> multichunks = list(genresult)
>>> depthprofile = ut.depth_profile(multichunks)
>>> assert depthprofile[1:] == chunksizes, 'did not generate chunks correctly'
>>> result = str(multichunks)
>>> print(result)
[[0, 1, 2], [3, 4, 5], [6, 0, 1]]
utool.util_iter.iter_window(iterable, size=2, step=1, wrap=False)[source]

iterates through iterable with a window size generalizeation of itertwo

Parameters:
  • iterable (iter) – an iterable sequence
  • size (int) – window size (default = 2)
  • wrap (bool) – wraparound (default = False)
Returns:

returns windows in a sequence
Return type:

iter

CommandLine:

python -m utool.util_iter --exec-iter_window

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> iterable = [1, 2, 3, 4, 5, 6]
>>> size, step, wrap = 3, 1, True
>>> window_iter = iter_window(iterable, size, step, wrap)
>>> window_list = list(window_iter)
>>> result = ('window_list = %r' % (window_list,))
>>> print(result)
window_list = [(1, 2, 3), (2, 3, 4), (3, 4, 5), (4, 5, 6), (5, 6, 1), (6, 1, 2)]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> iterable = [1, 2, 3, 4, 5, 6]
>>> size, step, wrap = 3, 2, True
>>> window_iter = iter_window(iterable, size, step, wrap)
>>> window_list = list(window_iter)
>>> result = ('window_list = %r' % (window_list,))
>>> print(result)
window_list = [(1, 2, 3), (3, 4, 5), (5, 6, 1)]
utool.util_iter.itertwo(iterable, wrap=False)[source]

equivalent to iter_window(iterable, 2, 1, wrap)

Parameters:
  • iterable (iter) – an iterable sequence
  • wrap (bool) – if True, returns with wraparound
Returns:

returns edges in a sequence
Return type:

iter

CommandLine:

python -m utool.util_iter --test-itertwo
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> iterable = [1, 2, 3, 4]
>>> wrap = False
>>> edges = list(itertwo(iterable, wrap))
>>> result = ('edges = %r' % (edges,))
>>> print(result)
edges = [(1, 2), (2, 3), (3, 4)]
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> iterable = [1, 2, 3, 4]
>>> wrap = True
>>> edges = list(itertwo(iterable, wrap))
>>> result = ('edges = %r' % (edges,))
>>> print(result)
edges = [(1, 2), (2, 3), (3, 4), (4, 1)]
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> import utool as ut
>>> iterable = iter([1, 2, 3, 4])
>>> wrap = False
>>> edge_iter = itertwo(iterable, wrap)
>>> edges = list(edge_iter)
>>> result = ('edges = %r' % (edges,))
>>> ut.assert_eq(len(list(iterable)), 0, 'iterable should have been used up')
>>> print(result)
edges = [(1, 2), (2, 3), (3, 4)]
utool.util_iter.next_counter(start=0, step=1)[source]
Parameters:
  • start (int) – (default = 0)
  • step (int) – (default = 1)
Returns:

next_
Return type:

func

CommandLine:

python -m utool.util_iter --test-next_counter

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_iter import *  # NOQA
>>> start = 1
>>> step = 1
>>> next_ = next_counter(start, step)
>>> result = str([next_(), next_(), next_()])
>>> print(result)
[1, 2, 3]
utool.util_iter.wrap_iterable(obj)[source]
Returns:wrapped_obj, was_scalar

utool.util_latex module

TODO: box and whisker http://tex.stackexchange.com/questions/115210/boxplot-in-latex

utool.util_latex.compile_latex_text(input_text, fnum=1, dpath=None, verbose=True, fname=None, title=None, nest_in_doc=None, **kwargs)[source]

pdflatex -shell-escape –synctex=-1 -src-specials -interaction=nonstopmode /home/joncrall/code/ibeis/tmptex/latex_formatter_temp.tex

CommandLine:

python -m utool.util_latex --test-compile_latex_text --show
Example1:
>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> import utool as ut
>>> verbose = True
>>> #dpath = '/home/joncrall/code/ibeis/aidchallenge'
>>> dpath = dirname(ut.grab_test_imgpath())
>>> #ut.vd(dpath)
>>> orig_fpath_list = ut.list_images(dpath, fullpath=True)
>>> figure_str = ut.get_latex_figure_str(orig_fpath_list, width_str='2.4in', nCols=2)
>>> input_text = figure_str
>>> pdf_fpath = ut.compile_latex_text(input_text, dpath=dpath, verbose=verbose)
>>> output_pdf_fpath = ut.compress_pdf(pdf_fpath)
>>> print(pdf_fpath)
>>> ut.quit_if_noshow()
>>> ut.startfile(pdf_fpath)

fpath_list def clipwhite_ondisk(fpath_in):

import utool as ut import vtool as vt fpath_out = ut.augpath(fpath_in, ‘_clipwhite’) img = vt.imread(fpath_in) thresh = 128 fillval = [255, 255, 255] cropped_img = vt.crop_out_imgfill(img, fillval=fillval, thresh=thresh) vt.imwrite(fpath_out, cropped_img) return fpath_out

fpath_list_ = [clipwhite_ondisk(fpath) for fpath in fpath_list] tmpfig = join(dpath, ‘tmpfig’) ut.ensuredir(tmpfig) # Weirdness from os.path import * new_fpath_list = [] for fpath in fpath_list_:

fname, ext = splitext(basename(fpath)) fname_ = ut.hashstr(fname, alphabet=ut.ALPHABET_16) + ext fpath_ = join(tmpfig, fname_) ut.move(fpath, fpath_) new_fpath_list.append(fpath_)

new_rel_fpath_list = [ut.relpath_unix(fpath_, dpath) for fpath_ in new_fpath_list]

utool.util_latex.compress_pdf(pdf_fpath, output_fname=None)[source]

uses ghostscript to write a pdf

utool.util_latex.ensure_colvec(arr)[source]
utool.util_latex.ensure_rowvec(arr)[source]
utool.util_latex.escape_latex(text)[source]
Parameters:text (str) – a plain text message
Returns:the message escaped to appear correctly in LaTeX
Return type:str

References

http://stackoverflow.com/questions/16259923/how-can-i-escape-latex-special-characters-inside-django-templates

utool.util_latex.find_ghostscript_exe()[source]
utool.util_latex.get_bibtex_dict(bib_fpath)[source]
Parameters:bib_fpath (str) –
Returns:bibtex_dict
Return type:dict

CommandLine:

python -m utool.util_latex --test-get_bibtex_dict
pip install bibtexparser

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> import utool as ut
>>> bib_fpath = ut.truepath('~/latex/crall-candidacy-2015/My_Library_clean.bib')
>>> bibtex_dict = get_bibtex_dict(bib_fpath)
>>> result = ('bibtex_dict = %s' % (str(bibtex_dict),))
>>> print(result)
utool.util_latex.get_latex_figure_str(fpath_list, caption_str=None, label_str=None, width_str='\\textwidth', height_str=None, nCols=None, dpath=None, colpos_sep=' ', nlsep='', use_sublbls=None, use_frame=False)[source]
Parameters:
  • fpath_list (list) –
  • dpath (str) – directory relative to main tex file
Returns:

figure_str
Return type:

str

CommandLine:

python -m utool.util_latex --test-get_latex_figure_str

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> # build test data
>>> fpath_list = ['figures/foo.png']
>>> # execute function
>>> figure_str = get_latex_figure_str(fpath_list)
>>> # verify results
>>> result = str(figure_str)
>>> print(result)
utool.util_latex.get_latex_figure_str2(fpath_list, cmdname, **kwargs)[source]

hack for candidacy

utool.util_latex.is_substr(find, strlist)[source]
utool.util_latex.latex_get_stats(lbl, data, mode=0)[source]
utool.util_latex.latex_multicolumn(data, ncol=2, alignstr='|c|')[source]
utool.util_latex.latex_multirow(data, nrow=2)[source]
utool.util_latex.latex_newcommand(command_name, command_text, num_args=0)[source]
utool.util_latex.latex_sanatize_command_name(_cmdname)[source]
Parameters:_cmdname –
Returns:command_name
Return type:?

CommandLine:

python -m utool.util_latex --exec-latex_sanatize_command_name

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> _cmdname = '#foo bar.'
>>> command_name = latex_sanatize_command_name(_cmdname)
>>> result = ('command_name = %s' % (str(command_name),))
>>> print(result)
FooBar
utool.util_latex.latex_scalar(lbl, data)[source]
utool.util_latex.long_substr(strlist)[source]
utool.util_latex.make_full_document(text, title=None, preamp_decl={})[source]

dummy preamble and document to wrap around latex fragment

Parameters:
  • text (str) –
  • title (str) –
Returns:

text_
Return type:

str

CommandLine:

python -m utool.util_latex --test-make_full_document

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> text = 'foo'
>>> title = 'title'
>>> preamp_decl = {}
>>> text_ = make_full_document(text, title)
>>> result = str(text_)
>>> print(result)
utool.util_latex.make_score_tabular(row_lbls, col_lbls, values, title=None, out_of=None, bold_best=False, flip=False, bigger_is_better=True, multicol_lbls=None, FORCE_INT=False, precision=None, SHORTEN_ROW_LBLS=False, col_align='l', col_sep='|', multicol_sep='|', centerline=True, astable=False, table_position='', AUTOFIX_LATEX=True, **kwargs)[source]

makes a LaTeX tabular for displaying scores or errors

Parameters:
  • row_lbls (list of str) –
  • col_lbls (list of str) –
  • values (ndarray) –
  • title (str) – (default = None)
  • out_of (None) – (default = None)
  • bold_best (bool) – (default = True)
  • flip (bool) – (default = False)
  • table_position (str) – eg ‘[h]’
Returns:

tabular_str
Return type:

str

CommandLine:

python -m utool.util_latex --test-make_score_tabular:0 --show
python -m utool.util_latex --test-make_score_tabular:1 --show
python -m utool.util_latex --test-make_score_tabular:2 --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> import utool as ut
>>> row_lbls = ['config1', 'config2']
>>> col_lbls = ['score \leq 1', 'metric2']
>>> values = np.array([[1.2, 2], [3.2, 4]])
>>> title = 'title'
>>> out_of = 10
>>> bold_best = True
>>> flip = False
>>> tabular_str = make_score_tabular(row_lbls, col_lbls, values, title, out_of, bold_best, flip)
>>> result = tabular_str
>>> print(result)
>>> ut.quit_if_noshow()
>>> render_latex_text(tabular_str)

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> import utool as ut
>>> row_lbls = ['config1']
>>> col_lbls = ['score \leq 1', 'metric2']
>>> values = np.array([[1.2, 2]])
>>> title = 'title'
>>> out_of = 10
>>> bold_best = True
>>> flip = False
>>> tabular_str = make_score_tabular(row_lbls, col_lbls, values, title, out_of, bold_best, flip)
>>> result = tabular_str
>>> print(result)
>>> ut.quit_if_noshow()
>>> render_latex_text(tabular_str)

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_latex import *  # NOQA
>>> import utool as ut
>>> row_lbls = ['config1', 'config2']
>>> col_lbls = ['score \leq 1', 'metric2', 'foobar']
>>> multicol_lbls = [('spam', 1), ('eggs', 2)]
>>> values = np.array([[1.2, 2, -3], [3.2, 4, -2]])
>>> title = 'title'
>>> out_of = 10
>>> bold_best = True
>>> flip = False
>>> tabular_str = make_score_tabular(row_lbls, col_lbls, values, title, out_of, bold_best, flip, multicol_lbls=multicol_lbls)
>>> result = tabular_str
>>> print(result)
>>> ut.quit_if_noshow()
>>> render_latex_text(tabular_str)
utool.util_latex.make_stats_tabular()[source]

tabular for dipslaying statistics

utool.util_latex.make_table2(row_lbls, col_lbls, values, **kwargs)[source]
utool.util_latex.render(input_text, fnum=1, dpath=None, verbose=True)[source]

fixme or remove

utool.util_latex.render_latex_text(input_text, nest_in_doc=False, appname='utool', verbose=None)[source]

testing function

utool.util_latex.replace_all(str_, repltups)[source]
utool.util_latex.tabular_join(tabular_body_list, nCols=2)[source]

utool.util_list module

utool.util_list.accumulate(iterator)[source]
Notice:
use itertools.accumulate in python > 3.2
utool.util_list.alloc_lists(num_alloc)[source]

allocates space for a list of lists

utool.util_list.alloc_nones(num_alloc)[source]

allocates space for a list of Nones

utool.util_list.allsame(list_, strict=True)[source]

checks to see if list is equal everywhere

Parameters:list (list) –
Returns:True if all items in the list are equal
utool.util_list.and_lists(*args)[source]
utool.util_list.argsort(*args, **kwargs)[source]

like np.argsort but for lists

Varargs:
*args (list): multiple lists to sort by
Kwargs:
reverse (bool): sort order is descending if True else acscending
utool.util_list.compress(item_list, flag_list)[source]

like np.compress but for lists

Returns items in item list where the corresponding item in flag list is True

Parameters:
  • item_list (list) – list of items to mask
  • flag_list (list) – list of booleans used as a mask
Returns:

filtered_items - masked items
Return type:

list
utool.util_list.debug_consec_list(list_)[source]
Returns:tuple of (missing_items, missing_indices, duplicate_items)
utool.util_list.debug_duplicate_items(items, *args, **kwargs)[source]
utool.util_list.delete_items_by_index(list_, index_list)[source]
Parameters:
  • list (list) –
  • index_list (list) –

CommandLine:

python -m utool.util_list --exec-delete_items_by_index

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [8, 1, 8, 1, 6, 6, 3, 4, 4, 5, 6]
>>> index_list = [2, -1]
>>> result = delete_items_by_index(list_, index_list)
>>> print(result)
[8, 1, 1, 6, 6, 3, 4, 4, 5]
utool.util_list.depth_profile(list_, max_depth=None, compress_homogenous=True, compress_consecutive=False, new_depth=False)[source]

Returns a nested list corresponding the shape of the nested structures lists represent depth, tuples represent shape. The values of the items do not matter. only the lengths.

Parameters:
  • list (list) –
  • max_depth (None) –
  • compress_homogenous (bool) –
  • compress_consecutive (bool) – experimental

CommandLine:

python -m utool.util_list --test-depth_profile
Setup:
>>> from utool.util_list import *  # NOQA
Example0:
>>> # ENABLE_DOCTEST
>>> list_ = [[[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]], [[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]]]
>>> result = depth_profile(list_)
>>> print(result)
(2, 3, 4)
Example1:
>>> # ENABLE_DOCTEST
>>> list_ = [[[[[1]]], [3, 4, 33]], [[1], [2, 3], [4, [5, 5]]], [1, 3]]
>>> result = depth_profile(list_)
>>> print(result)
[[(1, 1, 1), 3], [1, 2, [1, 2]], 2]
Example2:
>>> # ENABLE_DOCTEST
>>> list_ = [[[[[1]]], [3, 4, 33]], [[1], [2, 3], [4, [5, 5]]], [1, 3]]
>>> result = depth_profile(list_, max_depth=1)
>>> print(result)
[[(1, '1'), 3], [1, 2, [1, '2']], 2]
Example3:
>>> # ENABLE_DOCTEST
>>> list_ = [[[1, 2], [1, 2, 3]], None]
>>> result = depth_profile(list_, compress_homogenous=True)
>>> print(result)
[[2, 3], 1]
Example4:
>>> # ENABLE_DOCTEST
>>> list_ = [[3, 2], [3, 2], [3, 2], [3, 2], [3, 2], [3, 2], [9, 5, 3], [2, 2]]
>>> result = depth_profile(list_, compress_homogenous=True, compress_consecutive=True)
>>> print(result)
[2] * 6 + [3, 2]
Example5:
>>> # ENABLE_DOCTEST
>>> list_ = [[[3, 9], 2], [[3, 9], 2], [[3, 9], 2], [[3, 9], 2]]  #, [3, 2], [3, 2]]
>>> result = depth_profile(list_, compress_homogenous=True, compress_consecutive=True)
>>> print(result)
(4, [2, 1])
Example6:
>>> # ENABLE_DOCTEST
>>> list_ = [[[[1, 2]], [1, 2]], [[[1, 2]], [1, 2]], [[[0, 2]], [1]]]
>>> result1 = depth_profile(list_, compress_homogenous=True, compress_consecutive=False)
>>> result2 = depth_profile(list_, compress_homogenous=True, compress_consecutive=True)
>>> result = str(result1) + '\n' + str(result2)
>>> print(result)
[[(1, 2), 2], [(1, 2), 2], [(1, 2), 1]]
[[(1, 2), 2]] * 2 + [[(1, 2), 1]]
Example7:
>>> # ENABLE_DOCTEST
>>> list_ = [[{'a': [1, 2], 'b': [3, 4, 5]}, [1, 2, 3]], None]
>>> result = depth_profile(list_, compress_homogenous=True)
>>> print(result)
Example8:
>>> # ENABLE_DOCTEST
>>> list_ = [[[1]], [[[1, 1], [1, 1]]], [[[[1, 3], 1], [[1, 3, 3], 1, 1]]]]
>>> result = depth_profile(list_, compress_homogenous=True)
>>> print(result)
Example9:
>>> # ENABLE_DOCTEST
>>> list_ = []
>>> result = depth_profile(list_)
>>> print(result)

# THIS IS AN ERROR??? SHOULD BE #[1, 1], [1, 2, 2], (1, ([1, 2]), (

Example10:
>>> # ENABLE_DOCTEST
>>> fm1 = [[0, 0], [0, 0]]
>>> fm2 = [[0, 0], [0, 0], [0, 0]]
>>> fm3 = [[0, 0], [0, 0], [0, 0], [0, 0]]
>>> list_ = [0, 0, 0]
>>> list_ = [fm1, fm2, fm3]
>>> max_depth = 0
>>> new_depth = True
>>> result = depth_profile(list_, max_depth=max_depth, new_depth=new_depth)
>>> print(result)
utool.util_list.duplicates_exist(items)[source]

returns if list has duplicates

utool.util_list.ensure_list_size(list_, size_)[source]

Allocates more space if needbe.

Ensures len(list_) == size_.

Parameters:
  • list (list) – list to extend
  • size (int) – amount to exent by
utool.util_list.filter_Nones(item_list)[source]

Removes any nones from the list

Parameters:item_list (list) –
Returns:sublist which does not contain Nones
utool.util_list.filter_items(item_list, flag_list)[source]

Returns items in item list where the corresponding item in flag list is True

Parameters:
  • item_list (list) –
  • flag_list (list) –
Returns:

filtered_items
SeeAlso:
util_iter.iter_compress
utool.util_list.filter_startswith(list_, str_)[source]
utool.util_list.filterfalse_items(item_list, flag_list)[source]

Returns items in item list where the corresponding item in flag list is true

Parameters:
  • item_list (list) – list of items
  • flag_list (list) – list of truthy values
Returns:

items where the corresponding flag was truthy
Return type:

filtered_items
SeeAlso:
util_iter.ifilterfalse_items
utool.util_list.find_duplicate_items(items)[source]
Parameters:items (list) –
Returns:duplicate_map of indexes
Return type:dict

CommandLine:

python -m utool.util_list --test-find_duplicate_items

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> items = [0, 1, 2, 3, 3, 0, 12, 2, 9]
>>> duplicate_map = find_duplicate_items(items)
>>> result = str(duplicate_map)
>>> print(result)
utool.util_list.find_list_indexes(list_, item_list)[source]
Parameters:
  • list (list) – list of items to be searched
  • item_list (list) – list of items to find

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = ['a', 'b', 'c']
>>> item_list = ['d', 'c', 'b', 'f']
>>> index_list = find_list_indexes(list_, item_list)
>>> result = ('index_list = %r' % (index_list,))
>>> print(result)
index_list = [None, 2, 1, None]
utool.util_list.find_nonconsec_indices(unique_vals, consec_vals)[source]

# TODO: rectify with above function

Parameters:
  • unique_vals (list) –
  • consec_vals (list) –
Returns:

missing_ixs

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import numpy as np
>>> unique_vals = np.array([-2, -1,  1,  2, 10])
>>> max_ = unique_vals.max()
>>> min_ = unique_vals.min()
>>> range_ = max_ - min_
>>> consec_vals = np.linspace(min_, max_ + 1, range_ + 2)
>>> missing_ixs = find_nonconsec_indices(unique_vals, consec_vals)
>>> result = (consec_vals[missing_ixs])
[ 0.  3.  4.  5.  6.  7.  8.  9.]
utool.util_list.flag_None_items(list_)[source]
utool.util_list.flag_not_None_items(list_)[source]
utool.util_list.flag_unique_items(list_)[source]

Returns a list of flags corresponding to the first time an item is seen

Parameters:list (list) – list of items
Returns:flag_list
utool.util_list.flat_unique(*lists_, **kwargs)[source]

returns items unique across all lists

utool.util_list.flatten(list_)[source]
Parameters:list (list) – list of lists
Returns:flat list
Return type:list

CommandLine:

python -m utool.util_list --test-flatten

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> list_ = [['a', 'b'], ['c', 'd']]
>>> unflat_list2 = flatten(list_)
>>> result = ut.list_str(unflat_list2, nl=False)
>>> print(result)
['a', 'b', 'c', 'd']
utool.util_list.flattenize(list_)[source]

maps flatten to a tuplized list

Weird function. DEPRICATE

Example

>>> list_ = [[1, 2, 3], [2, 3, [4, 2, 1]], [3, 2], [[1, 2], [3, 4]]]
>>> import utool
>>> from itertools import zip
>>> val_list1 = [(1, 2), (2, 4), (5, 3)]
>>> id_list1  = [(1,),     (2,),   (3,)]
>>> out_list1 = utool.flattenize(zip(val_list1, id_list1))

>>> val_list2 = [1, 4, 5]
>>> id_list2  = [(1,),     (2,),   (3,)]
>>> out_list2 = utool.flattenize(zip(val_list2, id_list2))

>>> val_list3 = [1, 4, 5]
>>> id_list3  = [1, 2, 3]
>>> out_list3 = utool.flattenize(zip(val_list3, id_list3))

out_list4 = list(zip(val_list3, id_list3)) %timeit utool.flattenize(zip(val_list1, id_list1)) %timeit utool.flattenize(zip(val_list2, id_list2)) %timeit utool.flattenize(zip(val_list3, id_list3)) %timeit list(zip(val_list3, id_list3))

100000 loops, best of 3: 14 us per loop 100000 loops, best of 3: 16.5 us per loop 100000 loops, best of 3: 18 us per loop 1000000 loops, best of 3: 1.18 us per loop

utool.util_list.get_dirty_items(item_list, flag_list)[source]

Returns each item in item_list where not flag in flag_list

Parameters:
  • item_list (list) –
  • flag_list (list) –
Returns:

dirty_items
utool.util_list.get_list_column(list_, colx)[source]

accepts a list of (indexables) and returns a list of indexables can also return a list of list of indexables if colx is a list

Parameters:
  • list (list) – list of lists
  • colx (int or list) – index or key in each sublist get item
Returns:

list of selected items
Return type:

list

CommandLine:

python -m utool.util_list --test-get_list_column
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [['a', 'b'], ['c', 'd']]
>>> colx = 0
>>> result = get_list_column(list_, colx)
>>> import utool as ut
>>> result = ut.list_str(result, nl=False)
>>> print(result)
['a', 'c']
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [['a', 'b'], ['c', 'd']]
>>> colx = [1, 0]
>>> result = get_list_column(list_, colx)
>>> import utool as ut
>>> result = ut.list_str(result, nl=False)
>>> print(result)
[['b', 'a'], ['d', 'c']]
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [{'spam': 'EGGS', 'ham': 'SPAM'}, {'spam': 'JAM', 'ham': 'PRAM'},]
>>> # colx can be a key or list of keys as well
>>> colx = ['spam']
>>> result = get_list_column(list_, colx)
>>> import utool as ut
>>> result = ut.list_str(result, nl=False)
>>> print(result)
[['EGGS'], ['JAM']]
utool.util_list.get_list_column_slice(list_, start=None, stop=None, stride=None)[source]
utool.util_list.group_consecutives(data, stepsize=1)[source]

Return list of consecutive lists of numbers from data (number list).

References

http://stackoverflow.com/questions/7352684/how-to-find-the-groups-of-consecutive-elements-from-an-array-in-numpy

utool.util_list.group_consecutives_numpy(data, stepsize=1)[source]
Parameters:
  • data –
  • stepsize (int) –
Returns:

list of ndarrays
Return type:

list

References

http://stackoverflow.com/questions/7352684/how-to-find-the-groups-of-consecutive-elements-from-an-array-in-numpy

CommandLine:

python -m utool.util_list --test-group_consecutives

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> # build test data
>>> data = np.array([  0,   1,   2,   3,   4, 320, 636, 637, 638, 639])
>>> stepsize = 1
>>> # execute function
>>> result = group_consecutives(data, stepsize)
>>> # verify results
>>> print(result)
[array([0, 1, 2, 3, 4]), array([320]), array([636, 637, 638, 639])]
Timeit::
%timeit group_consecutives_numpy(data, stepsize) # 14.8 µs per loop %timeit group_consecutives(data, stepsize) # 4.47 µs per loop
utool.util_list.iflag_unique_items(list_)[source]

Returns a list of flags corresponding to the first time an item is seen

Parameters:list (list) – list of items
Returns:flag_iter
utool.util_list.index_complement(index_list, len_=None)[source]

Returns the other indicies in a list of length len_

utool.util_list.index_to_boolmask(index_list, maxval=None)[source]
Parameters:
  • index_list (list) –
  • maxval (None) – (default = None)
Kwargs:
maxval
Returns:mask
Return type:list
SeeAlso:
vt.index_to_boolmask numpy version

CommandLine:

python -m vtool.other --exec-index_to_boolmask

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> index_list = [0, 1, 4]
>>> maxval = 5
>>> mask = ut.index_to_boolmask(index_list, maxval)
>>> result = ('mask = %s' % (ut.repr2(mask, nl=0)))
>>> print(result)
mask = [True, True, False, False, True]
utool.util_list.intersect_ordered(list1, list2)[source]

returns list1 elements that are also in list2. preserves order of list1

intersect_ordered

Parameters:
  • list1 (list) –
  • list2 (list) –
Returns:

new_list
Return type:

list

Example

>>> from utool.util_list import *  # NOQA
>>> list1 = ['featweight_rowid', 'feature_rowid', 'config_rowid', 'featweight_forground_weight']
>>> list2 = [u'featweight_rowid']
>>> result = intersect_ordered(list1, list2)
>>> print(result)
['featweight_rowid']
utool.util_list.invertible_flatten(unflat_list)[source]

Flattens list but remember how to reconstruct the unflat list Returns flat list and the unflat list with indexes into the flat list

Parameters:unflat_list (list) – list of nested lists that we will flatten.
Returns:(flat_list, reverse_list)
Return type:tuple

CommandLine:

python -m utool.util_list --exec-invertible_flatten --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> unflat_list = [[1, 2, 3], [4, 5], [6, 6]]
>>> flat_list, reverse_list = invertible_flatten(unflat_list)
>>> result = ('flat_list = %s\n' % (ut.repr2(flat_list),))
>>> result += ('reverse_list = %s' % (ut.repr2(reverse_list),))
>>> print(result)
flat_list = [1, 2, 3, 4, 5, 6, 6]
reverse_list = [[0, 1, 2], [3, 4], [5, 6]]
utool.util_list.invertible_flatten2(unflat_list)[source]

An alternative to invertible_flatten which uses cumsum

Flattens list but remember how to reconstruct the unflat list Returns flat list and the unflat list with indexes into the flat list

Parameters:unflat_list (list) –
Returns:flat_list, cumlen_list
Return type:tuple

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool
>>> utool.util_list
>>> unflat_list = [[5], [2, 3, 12, 3, 3], [9], [13, 3], [5]]
>>> flat_list, cumlen_list = invertible_flatten2(unflat_list)
>>> unflat_list2 = unflatten2(flat_list, cumlen_list)
>>> assert unflat_list2 == unflat_list
>>> result = ((flat_list, cumlen_list))
>>> print(result)
([5, 2, 3, 12, 3, 3, 9, 13, 3, 5], [1, 6, 7, 9, 10])

TODO: This flatten is faster fix it to be used everywhere

Timeit:

unflat_list = [[random.random() for _ in range(int(random.random() * 1000))] for __ in range(200)] unflat_arrs = list(map(np.array, unflat_list))

%timeit invertible_flatten2(unflat_list) %timeit invertible_flatten2_numpy(unflat_list) %timeit invertible_flatten2_numpy(unflat_arrs)

SeeAlso:
invertible_flatten unflatten unflatten2
Timeits:
import utool unflat_list = aids_list1 flat_aids1, reverse_list = utool.invertible_flatten(unflat_list) flat_aids2, cumlen_list = utool.invertible_flatten2(unflat_list) unflat_list1 = utool.unflatten(flat_aids1, reverse_list) unflat_list2 = utool.unflatten2(flat_aids2, cumlen_list) assert list(map(list, unflat_list1)) == unflat_list2 print(utool.get_object_size_str(unflat_list, ‘unflat_list ‘)) print(utool.get_object_size_str(flat_aids1, ‘flat_aids1 ‘)) print(utool.get_object_size_str(flat_aids2, ‘flat_aids2 ‘)) print(utool.get_object_size_str(reverse_list, ‘reverse_list ‘)) print(utool.get_object_size_str(cumlen_list, ‘cumlen_list ‘)) print(utool.get_object_size_str(unflat_list1, ‘unflat_list1 ‘)) print(utool.get_object_size_str(unflat_list2, ‘unflat_list2 ‘)) print(‘Timings 1:) %timeit utool.invertible_flatten(unflat_list) %timeit utool.unflatten(flat_aids1, reverse_list) print(‘Timings 2:) %timeit utool.invertible_flatten2(unflat_list) %timeit utool.unflatten2(flat_aids2, cumlen_list)
utool.util_list.invertible_flatten2_numpy(unflat_arrs, axis=0)[source]

more numpy version

TODO: move to vtool

Parameters:unflat_arrs (list) – list of ndarrays
Returns:(flat_list, cumlen_list)
Return type:tuple

CommandLine:

python -m utool.util_list --test-invertible_flatten2_numpy

Example

>>> # ENABLE_DOCTET
>>> from utool.util_list import *  # NOQA
>>> unflat_arrs = [np.array([1, 2, 1]), np.array([5, 9]), np.array([4])]
>>> (flat_list, cumlen_list) = invertible_flatten2_numpy(unflat_arrs)
>>> result = str((flat_list, cumlen_list))
>>> print(result)
(array([1, 2, 1, 5, 9, 4]), array([3, 5, 6]))
utool.util_list.is_subset(list1, list2)
utool.util_list.is_subset_of_any(set_, other_sets)[source]

returns True if set_ is a subset of any set in other_sets

Parameters:
  • set (set) –
  • other_sets (list of sets) –
Returns:

flag
Return type:

bool

CommandLine:

python -m utool.util_list --test-is_subset_of_any

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> # build test data
>>> set_ = {1, 2}
>>> other_sets = [{1, 4}, {3, 2, 1}]
>>> # execute function
>>> result = is_subset_of_any(set_, other_sets)
>>> # verify results
>>> print(result)
True

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> # build test data
>>> set_ = {1, 2}
>>> other_sets = [{1, 4}, {3, 2}]
>>> # execute function
>>> result = is_subset_of_any(set_, other_sets)
>>> # verify results
>>> print(result)
False
utool.util_list.is_superset(list1, list2)
utool.util_list.isetdiff_flags(list1, list2)[source]

move to util_iter

utool.util_list.issorted(list_, op=<built-in function le>)[source]
Parameters:
  • list (list) –
  • op (builtin_function_or_method) –
Returns:

True if the list is sorted
Return type:

bool
utool.util_list.issubset(list1, list2)
utool.util_list.issuperset(list1, list2)
utool.util_list.isunique(items)[source]
utool.util_list.list_all_eq_to(list_, val, strict=True)[source]

checks to see if list is equal everywhere to a value

Parameters:
  • list (list) –
  • val – value to check against
Returns:

True if all items in the list are equal to val
utool.util_list.list_argmax(list_)[source]
utool.util_list.list_argmaxima(list_)[source]
Parameters:list (list) –
Returns:argmaxima
Return type:list

CommandLine:

python -m utool.util_list --exec-list_argmaxima

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = np.array([1, 2, 3, 3, 3, 2, 1])
>>> argmaxima = list_argmaxima(list_)
>>> result = ('argmaxima = %s' % (str(argmaxima),))
>>> print(result)
argmaxima = [2 3 4]
utool.util_list.list_argsort(*args, **kwargs)

like np.argsort but for lists

Varargs:
*args (list): multiple lists to sort by
Kwargs:
reverse (bool): sort order is descending if True else acscending
utool.util_list.list_compress(item_list, flag_list)

like np.compress but for lists

Returns items in item list where the corresponding item in flag list is True

Parameters:
  • item_list (list) – list of items to mask
  • flag_list (list) – list of booleans used as a mask
Returns:

filtered_items - masked items
Return type:

list
utool.util_list.list_compresstake(items_list, flags_list)[source]
utool.util_list.list_cover(list1, list2)[source]

returns boolean for each position in list1 if it is in list2

Parameters:
  • list1 (list) –
  • list2 (list) –
Returns:

incover_list - true where list1 intersects list2
Return type:

list

CommandLine:

python -m utool.util_list --test-list_cover

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> # build test data
>>> list1 = [1, 2, 3, 4, 5, 6]
>>> list2 = [2, 3, 6]
>>> # execute function
>>> incover_list = list_cover(list1, list2)
>>> # verify results
>>> result = str(incover_list)
>>> print(result)
[False, True, True, False, False, True]
utool.util_list.list_deep_types(list_)[source]

Returns all types in a deep list

utool.util_list.list_depth(list_, func=<built-in function max>, _depth=0)[source]

Returns the deepest level of nesting within a list of lists

Parameters:
  • list – a nested listlike object
  • func – depth aggregation strategy (defaults to max)
  • _depth – internal var

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [[[[[1]]], [3]], [[1], [3]], [[1], [3]]]
>>> result = (list_depth(list_, _depth=0))
>>> print(result)
utool.util_list.list_getattr(list_, attrname)[source]
utool.util_list.list_intersection(list1, list2)[source]
utool.util_list.list_inverse_take(list_, index_list)[source]
Parameters:
  • list (list) – list in sorted domain
  • index_list (list) – index list of the unsorted domain

Note

Seems to be logically equivalent to ut.take(list_, ut.argsort(index_list)), but faster

Returns:output_list_ - the input list in the unsorted domain
Return type:list

CommandLine:

python -m utool.util_list --test-list_inverse_take

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> rank_list = [3, 2, 4, 1, 9, 2]
>>> prop_list = [0, 1, 2, 3, 4, 5]
>>> index_list = ut.argsort(rank_list)
>>> sorted_prop_list = ut.take(prop_list, index_list)
>>> # execute function
>>> list_ = sorted_prop_list
>>> output_list_  = list_inverse_take(list_, index_list)
>>> output_list2_ = ut.take(list_, ut.argsort(index_list))
>>> assert output_list_ == prop_list
>>> assert output_list2_ == prop_list
>>> # verify results
>>> result = str(output_list_)
>>> print(result)
Timeit::
%timeit list_inverse_take(list_, index_list) %timeit ut.take(list_, ut.argsort(index_list))
utool.util_list.list_issubset(list1, list2)[source]
utool.util_list.list_issuperset(list1, list2)[source]
utool.util_list.list_replace(list_, target, repl)[source]

alias

Recursively removes target in all lists and sublists and replaces them with the repl variable

utool.util_list.list_reshape(list_, new_shape, trail=False)[source]

reshapes leaving trailing dimnsions in front if prod(new_shape) != len(list_)

Parameters:
  • list (list) –
  • new_shape (tuple) –
Returns:

list_
Return type:

list

CommandLine:

python -m utool.util_list --exec-list_reshape --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> import numpy as np
>>> list_ = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
>>> new_shape = (2, 2, 3)
>>> newlist = list_reshape(list_, new_shape)
>>> depth = ut.depth_profile(newlist)
>>> result = ('list_ = %s' % (ut.repr2(newlist, nl=1),))
>>> print('depth = %r' % (depth,))
>>> print(result)
>>> newlist2 = np.reshape(list_, depth).tolist()
>>> ut.assert_eq(newlist, newlist2)
utool.util_list.list_roll(list_, n)[source]

Like numpy.roll for python lists

Parameters:
  • list (list) –
  • n (int) –
Returns:
Return type:

list

References

http://stackoverflow.com/questions/9457832/python-list-rotation

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [1, 2, 3, 4, 5]
>>> n = 2
>>> result = list_roll(list_, n)
>>> print(result)
[4, 5, 1, 2, 3]
utool.util_list.list_set_equal(list1, list2)[source]
utool.util_list.list_take(list_, index_list)

like np.take but for lists

Parameters:
  • list (list) – some indexable object
  • index_list (list, slice, int) – some indexing object
Returns:

subset of the list
Return type:

list or scalar

CommandLine:

python -m utool.util_list --test-take

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [0, 1, 2, 3]
>>> index_list = [2, 0]
>>> result = take(list_, index_list)
>>> print(result)
[2, 0]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [0, 1, 2, 3]
>>> index = 2
>>> result = take(list_, index)
>>> print(result)
2

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [0, 1, 2, 3]
>>> index = slice(1, None, 2)
>>> result = take(list_, index)
>>> print(result)
[1, 3]
utool.util_list.list_transpose(list_)[source]
Parameters:list (list) –
Returns:
Return type:list

CommandLine:

python -m utool.util_list --test-list_transpose

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [[1, 2], [3, 4]]
>>> result = list_transpose(list_)
>>> print(result)
[(1, 3), (2, 4)]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [[1, 2, 3], [3, 4]]
>>> try:
>>>     result = list_transpose(list_)
>>> except AssertionError:
>>>     pass
>>> else:
>>>     assert False, 'should error'
utool.util_list.list_type(list_)[source]
utool.util_list.list_type_profile(sequence, compress_homogenous=True, with_dtype=True)[source]

similar to depth_profile but reports types

Parameters:
  • sequence –
  • compress_homogenous (bool) – (default = True)
Kwargs:
compress_homogenous
Returns:level_type_str
Return type:str

CommandLine:

python -m utool.util_list --exec-list_type_profile

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> sequence = []
>>> compress_homogenous = True
>>> level_type_str = list_type_profile(sequence, compress_homogenous)
>>> result = ('level_type_str = %s' % (str(level_type_str),))
>>> print(result)
utool.util_list.list_unflat_take(items_list, unflat_index_list)[source]
Parameters:
  • items_list (list) –
  • unflat_index_list (list) –

CommandLine:

python -m utool.util_list --exec-list_unflat_take

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> items_list = [1, 2, 3, 4, 5]
>>> unflat_index_list = [[0, 1], [2, 3], [0, 4]]
>>> result = list_unflat_take(items_list, unflat_index_list)
>>> print(result)
[[1, 2], [3, 4], [1, 5]]
utool.util_list.list_where(flag_list)

takes flags returns indexes of True values

utool.util_list.list_zipcompress(items_list, flags_list)
SeeAlso:
vt.zipcompress
utool.util_list.list_zipflatten(*items_lists)[source]
utool.util_list.list_ziptake(items_list, indexes_list)
SeeAlso:
vt.ziptake
utool.util_list.listclip(list_, num, fromback=False)[source]
Parameters:
  • list (list) –
  • num (int) –
Returns:
Return type:

sublist

CommandLine:

python -m utool.util_list --test-listclip
Example1:
>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> # build test data
>>> list_ = [1, 2, 3, 4, 5]
>>> result_list = []
>>> # execute function
>>> num = 3
>>> result_list += [ut.listclip(list_, num)]
>>> num = 9
>>> result_list += [ut.listclip(list_, num)]
>>> # verify results
>>> result = ut.list_str(result_list)
>>> print(result)
[
    [1, 2, 3],
    [1, 2, 3, 4, 5],
]
Example2:
>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> # build test data
>>> list_ = [1, 2, 3, 4, 5]
>>> result_list = []
>>> # execute function
>>> num = 3
>>> result = ut.listclip(list_, num, fromback=True)
>>> print(result)
[3, 4, 5]
utool.util_list.listfind(list_, tofind)[source]

get the position of item tofind in list_ if it exists otherwise returns None

Parameters:
  • list –
  • tofind –
Returns:

index of tofind in list_
Return type:

int
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = ['a', 'b', 'c']
>>> tofind = 'b'
>>> result = listfind(list_, tofind)
>>> print(result)
1
utool.util_list.lmap(func, iter_, **kwargs)[source]

list map - eagerly evaulates map like in python2 (but you aren’t using that right?)

utool.util_list.make_index_lookup(list_)[source]
Parameters:list (list) – assumed to have unique items
Returns:mapping from item to index
Return type:dict

CommandLine:

python -m utool.util_list --exec-make_index_lookup

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> list_ = [5, 3, 8, 2]
>>> idx2_item = ut.make_index_lookup(list_)
>>> result = ut.dict_str(idx2_item, nl=False)
>>> assert ut.dict_take(idx2_item, list_) == list(range(len(list_)))
>>> print(result)
{2: 3, 3: 1, 5: 0, 8: 2}
utool.util_list.make_sortby_func(item_list, reverse=False)[source]
utool.util_list.maplen(iter_)[source]
utool.util_list.multi_replace(instr, search_list=[], repl_list=None)[source]

Does a string replace with a list of search and replacements

TODO: rename

utool.util_list.not_list(flag_list)[source]
utool.util_list.or_lists(*args)[source]
utool.util_list.partial_imap_1to1(func, si_func)[source]

a bit messy

utool.util_list.print_duplicate_map(duplicate_map, *args, **kwargs)[source]
utool.util_list.priority_argsort(list_, priority)[source]
Parameters:
  • list (list) –
  • priority (list) – desired order of items
Returns:

reordered_index_list
Return type:

list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> list_ = [2, 4, 6, 8, 10]
>>> priority = [8, 2, 6, 9]
>>> sortx = priority_argsort(list_, priority)
>>> reordered_list = priority_sort(list_, priority)
>>> assert ut.take(list_, sortx) == reordered_list
>>> result = str(sortx)
>>> print(result)
[3, 0, 2, 1, 4]
utool.util_list.priority_sort(list_, priority)[source]
Parameters:
  • list (list) –
  • priority (list) – desired order of items
Returns:

reordered_list
Return type:

list

CommandLine:

python -m utool.util_list --test-priority_argsort

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [2, 4, 6, 8, 10]
>>> priority = [8, 2, 6, 9]
>>> reordered_list = priority_sort(list_, priority)
>>> result = str(reordered_list)
>>> print(result)
[8, 2, 6, 4, 10]
utool.util_list.recursive_replace(list_, target, repl=-1)[source]

Recursively removes target in all lists and sublists and replaces them with the repl variable

utool.util_list.replace_nones(list_, repl=-1)[source]

Recursively removes Nones in all lists and sublists and replaces them with the repl variable

Parameters:
  • list (list) –
  • repl (obj) – replacement value
Returns:

list

CommandLine:

python -m utool.util_list --test-replace_nones

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> # build test data
>>> list_ = [None, 0, 1, 2]
>>> repl = -1
>>> # execute function
>>> repl_list = replace_nones(list_, repl)
>>> # verify results
>>> result = str(repl_list)
>>> print(result)
[-1, 0, 1, 2]
utool.util_list.safe_listget(list_, index, default=u'?')[source]

depricate

utool.util_list.safe_slice(list_, [start, ]stop[, end][, step])[source]

Slices list and truncates if out of bounds

utool.util_list.sample_lists(items_list, num=1, seed=None)[source]
Parameters:
  • items_list (list) –
  • num (int) – (default = 1)
  • seed (None) – (default = None)
Returns:

samples_list
Return type:

list

CommandLine:

python -m utool.util_list --exec-sample_lists

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> items_list = [[], [1, 2, 3], [4], [5, 6], [7, 8, 9, 10]]
>>> num = 2
>>> seed = 0
>>> samples_list = sample_lists(items_list, num, seed)
>>> result = ('samples_list = %s' % (str(samples_list),))
>>> print(result)
samples_list = [[], [3, 2], [4], [5, 6], [10, 9]]
utool.util_list.sample_zip(items_list, num_samples, allow_overflow=False, per_bin=1)[source]

Helper for sampling

Given a list of lists, samples one item for each list and bins them into num_samples bins. If all sublists are of equal size this is equivilent to a zip, but otherewise consecutive bins will have monotonically less elemements

# Doctest doesn’t work with assertionerror #util_list.sample_zip(items_list, 2) #... #AssertionError: Overflow occured

Parameters:
  • items_list (list) –
  • num_samples –
  • allow_overflow (bool) –
  • per_bin (int) –
Returns:

(samples_list, overflow_samples)
Return type:

tuple

Examples

>>> from utool import util_list
>>> items_list = [[1, 2, 3, 4, 0], [5, 6, 7], [], [8, 9], [10]]
>>> util_list.sample_zip(items_list, 5)
...
[[1, 5, 8, 10], [2, 6, 9], [3, 7], [4], [0]]
>>> util_list.sample_zip(items_list, 2, allow_overflow=True)
...
([[1, 5, 8, 10], [2, 6, 9]], [3, 7, 4])
>>> util_list.sample_zip(items_list, 4, allow_overflow=True, per_bin=2)
...
([[1, 5, 8, 10, 2, 6, 9], [3, 7, 4], [], []], [0])
utool.util_list.scalar_input_map(func, input_)[source]

Map like function

Parameters:
  • func – function to apply
  • input – either an iterable or scalar value
Returns:

If input_ is iterable this function behaves like map otherwise applies func to input_
utool.util_list.search_list(text_list, pattern, flags=0)[source]

CommandLine:

python -m utool.util_list --test-search_list

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> text_list = ['ham', 'jam', 'eggs', 'spam']
>>> pattern = '.am'
>>> flags = 0
>>> (valid_index_list, valid_match_list) = ut.search_list(text_list, pattern, flags)
>>> result = str(valid_index_list)
>>> print(result)
[0, 1, 3]
utool.util_list.setdiff(list1, list2)[source]

returns list1 elements that are not in list2. preserves order of list1

Parameters:
  • list1 (list) –
  • list2 (list) –
Returns:

new_list
Return type:

list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> list1 = ['featweight_rowid', 'feature_rowid', 'config_rowid', 'featweight_forground_weight']
>>> list2 = [u'featweight_rowid']
>>> new_list = setdiff_ordered(list1, list2)
>>> result = ut.list_str(new_list, nl=False)
>>> print(result)
['feature_rowid', 'config_rowid', 'featweight_forground_weight']
utool.util_list.setdiff_flags(list1, list2)[source]
utool.util_list.setdiff_ordered(list1, list2)

returns list1 elements that are not in list2. preserves order of list1

Parameters:
  • list1 (list) –
  • list2 (list) –
Returns:

new_list
Return type:

list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> list1 = ['featweight_rowid', 'feature_rowid', 'config_rowid', 'featweight_forground_weight']
>>> list2 = [u'featweight_rowid']
>>> new_list = setdiff_ordered(list1, list2)
>>> result = ut.list_str(new_list, nl=False)
>>> print(result)
['feature_rowid', 'config_rowid', 'featweight_forground_weight']
utool.util_list.setintersect(list1, list2)

returns list1 elements that are in list2. preserves order of list1

setintersect_ordered

Parameters:
  • list1 (list) –
  • list2 (list) –
Returns:

new_list
Return type:

list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list1 = [1, 2, 3, 5, 8, 13, 21]
>>> list2 = [6, 4, 2, 21, 8]
>>> new_list = setintersect_ordered(list1, list2)
>>> result = new_list
>>> print(result)
[2, 8, 21]
utool.util_list.setintersect_ordered(list1, list2)[source]

returns list1 elements that are in list2. preserves order of list1

setintersect_ordered

Parameters:
  • list1 (list) –
  • list2 (list) –
Returns:

new_list
Return type:

list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list1 = [1, 2, 3, 5, 8, 13, 21]
>>> list2 = [6, 4, 2, 21, 8]
>>> new_list = setintersect_ordered(list1, list2)
>>> result = new_list
>>> print(result)
[2, 8, 21]
utool.util_list.sortedby(item_list, key_list, reverse=False)[source]

sorts item_list using key_list

Parameters:
  • list (list) – list to sort
  • key_list (list) – list to sort by
  • reverse (bool) – sort order is descending (largest first) if reverse is True else acscending (smallest first)
Returns:

list_ sorted by the values of another list. defaults to ascending order
Return type:

list
SeeAlso:
sortedby2

Examples

>>> # ENABLE_DOCTEST
>>> import utool
>>> list_    = [1, 2, 3, 4, 5]
>>> key_list = [2, 5, 3, 1, 5]
>>> result = utool.sortedby(list_, key_list, reverse=True)
[5, 2, 3, 1, 4]
utool.util_list.sortedby2(item_list, *args, **kwargs)[source]

sorts item_list using key_list

Parameters:item_list (list) – list to sort
Varargs:
*args (list): multiple lists to sort by
Kwargs:
reverse (bool): sort order is descending if True else acscending
Returns:list_ sorted by the values of another list. defaults to ascending order
Return type:list

CommandLine:

python -m utool.util_list --exec-sortedby2 --show

Examples

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> item_list = [1, 2, 3, 4, 5]
>>> key_list1 = [1, 1, 2, 3, 4]
>>> key_list2 = [2, 1, 4, 1, 1]
>>> args = (key_list1, key_list2)
>>> kwargs = dict(reverse=False)
>>> result = ut.sortedby2(item_list, *args, **kwargs)
>>> print(result)
[2, 1, 3, 4, 5]

Examples

>>> # ENABLE_DOCTEST
>>> # Python 3 Compatibility Test
>>> import utool as ut
>>> item_list = [1, 2, 3, 4, 5]
>>> key_list1 = ['a', 'a', 2, 3, 4]
>>> key_list2 = ['b', 'a', 4, 1, 1]
>>> args = (key_list1, key_list2)
>>> kwargs = dict(reverse=False)
>>> result = ut.sortedby2(item_list, *args, **kwargs)
>>> print(result)
[3, 4, 5, 2, 1]
utool.util_list.strided_sample(items, num, offset=0)[source]

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> # build test data
>>> items = [1, 2, 3, 4, 5]
>>> num = 3
>>> offset = 0
>>> # execute function
>>> sample_items = strided_sample(items, num, offset)
>>> # verify results
>>> result = str(sample_items)
>>> print(result)
utool.util_list.take(list_, index_list)[source]

like np.take but for lists

Parameters:
  • list (list) – some indexable object
  • index_list (list, slice, int) – some indexing object
Returns:

subset of the list
Return type:

list or scalar

CommandLine:

python -m utool.util_list --test-take

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [0, 1, 2, 3]
>>> index_list = [2, 0]
>>> result = take(list_, index_list)
>>> print(result)
[2, 0]

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [0, 1, 2, 3]
>>> index = 2
>>> result = take(list_, index)
>>> print(result)
2

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [0, 1, 2, 3]
>>> index = slice(1, None, 2)
>>> result = take(list_, index)
>>> print(result)
[1, 3]
utool.util_list.take_column(list_, colx)

accepts a list of (indexables) and returns a list of indexables can also return a list of list of indexables if colx is a list

Parameters:
  • list (list) – list of lists
  • colx (int or list) – index or key in each sublist get item
Returns:

list of selected items
Return type:

list

CommandLine:

python -m utool.util_list --test-get_list_column
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [['a', 'b'], ['c', 'd']]
>>> colx = 0
>>> result = get_list_column(list_, colx)
>>> import utool as ut
>>> result = ut.list_str(result, nl=False)
>>> print(result)
['a', 'c']
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [['a', 'b'], ['c', 'd']]
>>> colx = [1, 0]
>>> result = get_list_column(list_, colx)
>>> import utool as ut
>>> result = ut.list_str(result, nl=False)
>>> print(result)
[['b', 'a'], ['d', 'c']]
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [{'spam': 'EGGS', 'ham': 'SPAM'}, {'spam': 'JAM', 'ham': 'PRAM'},]
>>> # colx can be a key or list of keys as well
>>> colx = ['spam']
>>> result = get_list_column(list_, colx)
>>> import utool as ut
>>> result = ut.list_str(result, nl=False)
>>> print(result)
[['EGGS'], ['JAM']]
utool.util_list.take_complement(list_, index_list)[source]

Returns items in list_ not indexed by index_list

utool.util_list.take_percentile(arr, percent)[source]

take the top percent items in a list rounding up

utool.util_list.tuplize(list_)[source]

Converts each scalar item in a list to a dimension-1 tuple

utool.util_list.type_profile(sequence, compress_homogenous=True, with_dtype=True)

similar to depth_profile but reports types

Parameters:
  • sequence –
  • compress_homogenous (bool) – (default = True)
Kwargs:
compress_homogenous
Returns:level_type_str
Return type:str

CommandLine:

python -m utool.util_list --exec-list_type_profile

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> sequence = []
>>> compress_homogenous = True
>>> level_type_str = list_type_profile(sequence, compress_homogenous)
>>> result = ('level_type_str = %s' % (str(level_type_str),))
>>> print(result)
utool.util_list.unflat_map(func, unflat_items, vectorized=False, **kwargs)[source]

Uses an ibeis lookup function with a non-flat rowid list. In essence this is equivilent to [list(map(func, _items)) for _items in unflat_items]. The utility of this function is that it only calls method once. This is more efficient for calls that can take a list of inputs

Parameters:
  • func (func) – function
  • unflat_items (list) – list of rowid lists
Returns:

unflat_vals
Return type:

list of values

CommandLine:

python -m utool.util_list --test-unflat_map

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> vectorized = False
>>> kwargs = {}
>>> func = lambda x: x + 1
>>> unflat_items = [[], [1, 2, 3], [4, 5], [6, 7, 8, 9], [], []]
>>> unflat_vals = unflat_map(func, unflat_items)
>>> result = str(unflat_vals)
>>> print(result)
[[], [2, 3, 4], [5, 6], [7, 8, 9, 10], [], []]
utool.util_list.unflat_unique_rowid_map(func, unflat_rowids, **kwargs)[source]

performs only one call to the underlying func with unique rowids the func must be some lookup function

TODO: move this to a better place.

CommandLine:

python -m utool.util_list --test-unflat_unique_rowid_map:0
python -m utool.util_list --test-unflat_unique_rowid_map:1
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> kwargs = {}
>>> unflat_rowids = [[1, 2, 3], [2, 5], [1], []]
>>> num_calls0 = [0]
>>> num_input0 = [0]
>>> def func0(rowids, num_calls0=num_calls0, num_input0=num_input0):
...    num_calls0[0] += 1
...    num_input0[0] += len(rowids)
...    return [rowid + 10 for rowid in rowids]
>>> func = func0
>>> unflat_vals = unflat_unique_rowid_map(func, unflat_rowids, **kwargs)
>>> result = [arr.tolist() for arr in unflat_vals]
>>> print(result)
>>> ut.assert_eq(num_calls0[0], 1)
>>> ut.assert_eq(num_input0[0], 4)
[[11, 12, 13], [12, 15], [11], []]
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool as ut
>>> import numpy as np
>>> kwargs = {}
>>> unflat_rowids = [[1, 2, 3], [2, 5], [1], []]
>>> num_calls1 = [0]
>>> num_input1 = [0]
>>> def func1(rowids, num_calls1=num_calls1, num_input1=num_input1, np=np):
...    num_calls1[0] += 1
...    num_input1[0] += len(rowids)
...    return [np.array([rowid + 10, rowid, 3]) for rowid in rowids]
>>> func = func1
>>> unflat_vals = unflat_unique_rowid_map(func, unflat_rowids, **kwargs)
>>> result = [arr.tolist() for arr in unflat_vals]
>>> print(result)
>>> ut.assert_eq(num_calls1[0], 1)
>>> ut.assert_eq(num_input1[0], 4)
[[[11, 1, 3], [12, 2, 3], [13, 3, 3]], [[12, 2, 3], [15, 5, 3]], [[11, 1, 3]], []]
utool.util_list.unflat_vecmap(func, unflat_items, vectorized=False, **kwargs)[source]

unflat map for vectorized functions

utool.util_list.unflatten(flat_list, reverse_list)[source]

Rebuilds unflat list from invertible_flatten

Parameters:
  • flat_list (list) – the flattened list
  • reverse_list (list) – the list which undoes flattenting
Returns:

original nested list
Return type:

unflat_list2
SeeAlso:
invertible_flatten invertible_flatten2 unflatten2
utool.util_list.unflatten2(flat_list, cumlen_list)[source]

Rebuilds unflat list from invertible_flatten

Parameters:
  • flat_list (list) – the flattened list
  • cumlen_list (list) – the list which undoes flattenting
Returns:

original nested list
Return type:

unflat_list2
SeeAlso:
invertible_flatten invertible_flatten2 unflatten2

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> import utool
>>> utool.util_list
>>> flat_list = [5, 2, 3, 12, 3, 3, 9, 13, 3, 5]
>>> cumlen_list = [ 1,  6,  7,  9, 10]
>>> unflat_list2 = unflatten2(flat_list, cumlen_list)
>>> result = (unflat_list2)
>>> print(result)
[[5], [2, 3, 12, 3, 3], [9], [13, 3], [5]]
utool.util_list.unique(list_, ordered=True)[source]

Returns unique items in list_. Generally, unordered (*should be) faster.

utool.util_list.unique_ordered(list_)[source]

Returns unique items in list_ in the order they were seen.

Parameters:list (list) –
Returns:unique_list - unique list which maintains order
Return type:list

CommandLine:

python -m utool.util_list --exec-unique_ordered

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> list_ = [4, 6, 6, 0, 6, 1, 0, 2, 2, 1]
>>> unique_list = unique_ordered(list_)
>>> result = ('unique_list = %s' % (str(unique_list),))
>>> print(result)
unique_list = [4, 6, 0, 1, 2]
utool.util_list.unique_unordered(list_)[source]

wrapper around list(set(list_))

utool.util_list.where(flag_list)[source]

takes flags returns indexes of True values

utool.util_list.where_not_None(item_list)[source]

returns list of indexes of non None values

SeeAlso:
flag_None_items
utool.util_list.xor_lists(*args)[source]
Returns:
Return type:list

CommandLine:

python -m utool.util_list --test-xor_lists

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_list import *  # NOQA
>>> args = ([True, False, False, True], [True, True, False, False])
>>> result = xor_lists(*args)
>>> print(result)
[False, True, False, True]
utool.util_list.zipcompress(items_list, flags_list)[source]
SeeAlso:
vt.zipcompress
utool.util_list.ziptake(items_list, indexes_list)[source]
SeeAlso:
vt.ziptake

utool.util_logging module

If logging is on, utool will overwrite the print function with a logging function

This is a special module which will not get injected into (should it be internal?)

References

# maybe we can do something like this Queue to try fixing error when # when using injected print statments with Qt signals and slots http://stackoverflow.com/questions/21071448/redirecting-stdout-and-stderr-to-a-pyqt4-qtextedit-from-a-secondary-thread

utool.util_logging.__PYTHON_FLUSH__()

flush() -> None. Flush the internal I/O buffer.

utool.util_logging.__PYTHON_WRITE__()

write(str) -> None. Write string str to file.

Note that due to buffering, flush() or close() may be needed before the file on disk reflects the data written.

utool.util_logging.__UTOOL_FLUSH__()

flush() -> None. Flush the internal I/O buffer.

utool.util_logging.__UTOOL_WRITE__()

write(str) -> None. Write string str to file.

Note that due to buffering, flush() or close() may be needed before the file on disk reflects the data written.

utool.util_logging.add_logging_handler(handler, format_=u'file')[source]

mostly for util_logging internals

utool.util_logging.get_current_log_fpath()[source]
utool.util_logging.get_current_log_text()[source]
utool.util_logging.get_log_fpath(num=u'next', appname=None, log_dir=None)[source]
Returns:log_fpath – path to log file
Return type:str
utool.util_logging.get_logging_dir(appname=u'default')[source]

The default log dir is in the system resource directory But the utool global cache allows for the user to override where the logs for a specific app should be stored.

Returns:log_dir_realpath – real path to logging directory
Return type:str
utool.util_logging.get_utool_logger()[source]
utool.util_logging.start_logging(log_fpath=None, mode=u'a', appname=u'default', log_dir=None)[source]

Overwrites utool print functions to use a logger

CommandLine:

python -m utool.util_logging --test-start_logging:0
python -m utool.util_logging --test-start_logging:1
Example0:
>>> # DISABLE_DOCTEST
>>> import sys
>>> sys.argv.append('--verb-logging')
>>> import utool as ut
>>> ut.start_logging()
>>> ut.util_logging.__UTOOL_PRINT__('hello world')
>>> ut.util_logging.__UTOOL_WRITE__('writing1')
>>> ut.util_logging.__UTOOL_WRITE__('writing2\n')
>>> ut.util_logging.__UTOOL_WRITE__('writing3')
>>> ut.util_logging.__UTOOL_FLUSH__()
>>> handler = ut.util_logging.__UTOOL_ROOT_LOGGER__.handlers[0]
>>> current_log_fpath = handler.stream.name
>>> current_log_text = ut.read_from(current_log_fpath)
>>> assert current_log_text.find('hello world') > 0
>>> assert current_log_text.find('writing1writing2') > 0
>>> assert current_log_text.find('writing3') > 0
Example1:
>>> # DISABLE_DOCTEST
>>> # Ensure that progress is logged
>>> import sys
>>> sys.argv.append('--verb-logging')
>>> import utool as ut
>>> ut.start_logging()
>>> [x for x in  ut.ProgressIter(range(0, 1000), freq=4)]
>>> handler = ut.util_logging.__UTOOL_ROOT_LOGGER__.handlers[0]
>>> current_log_fpath = handler.stream.name
>>> current_log_text = ut.read_from(current_log_fpath)
>>> assert current_log_text.find('rate') > 0, 'progress was not logged'
>>> print(current_log_text)
utool.util_logging.stop_logging()[source]

Restores utool print functions to python defaults

utool.util_num module

utool.util_num.float_to_decimal(f)[source]

Convert a floating point number to a Decimal with no loss of information

References

http://docs.python.org/library/decimal.html#decimal-faq

utool.util_num.get_sys_maxfloat()[source]
utool.util_num.get_sys_maxint()[source]
utool.util_num.get_sys_minint()[source]
utool.util_num.int_comma_str(num)[source]
utool.util_num.num2_sigfig(num)[source]
utool.util_num.num_fmt(num, max_digits=None)[source]

Weird function. Not very well written. Very special case-y

Parameters:
  • num (int or float) –
  • max_digits (int) –
Returns:
Return type:

str

CommandLine:

python -m utool.util_num --test-num_fmt

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_num import *  # NOQA
>>> # build test data
>>> num_list = [0, 0.0, 1.2, 1003232, 41431232., .0000000343, -.443243]
>>> max_digits = None
>>> # execute function
>>> result = [num_fmt(num, max_digits) for num in num_list]
>>> # verify results
>>> print(result)
['0', '0.0', '1.2', '1,003,232', '41431232.0', '0.0', '-0.443']
utool.util_num.order_of_magnitude_ceil(num)[source]
utool.util_num.sigfig_str(number, sigfig)[source]

References

http://stackoverflow.com/questions/2663612/nicely-repr-float-in-python

utool.util_numpy module

utool.util_numpy.deterministic_sample(list_, nSample, seed=None, rng=None, strict=False)[source]

Grabs data randomly, but in a repeatable way

utool.util_numpy.deterministic_shuffle(list_, seed=None, rng=None)[source]
Parameters:
  • list (list) –
  • seed (int) –
Returns:

list_
Return type:

list

CommandLine:

python -m utool.util_numpy --test-deterministic_shuffle

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_numpy import *  # NOQA
>>> list_ = [1, 2, 3, 4, 5, 6]
>>> seed = 1
>>> list_ = deterministic_shuffle(list_, seed)
>>> result = str(list_)
>>> print(result)
[3, 2, 5, 1, 4, 6]
utool.util_numpy.inbounds(arr, low, high)[source]
utool.util_numpy.index_of(item, array)[source]

index of [item] in [array]

utool.util_numpy.intersect2d(A, B)[source]

intersect rows of 2d numpy arrays

DEPRICATE: use intersect2d in vtool instead

Parameters:
  • A (ndarray[ndim=2]) –
  • B (ndarray[ndim=2]) –
Returns:

(C, Ax, Bx)
Return type:

tuple

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_numpy import *  # NOQA
>>> import utool as ut
>>> A = np.array([[1, 2, 3], [1, 1, 1]])
>>> B = np.array([[1, 2, 3], [1, 2, 14]])
>>> (C, Ax, Bx) = ut.intersect2d(A, B)
>>> result = str((C, Ax, Bx))
>>> print(result)
(array([[1, 2, 3]]), array([0]), array([0]))
utool.util_numpy.make_incrementer()[source]
utool.util_numpy.npfind(arr)[source]
utool.util_numpy.random_indexes(max_index, subset_size=None, seed=None, rng=None)[source]

random unrepeated indicies

Parameters:
  • max_index –
  • subset_size (None) – (default = None)
  • seed (None) – (default = None)
  • rng (RandomState) – random number generator(default = None)
Returns:

subst
Return type:

?

CommandLine:

python -m utool.util_numpy --exec-random_indexes

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_numpy import *  # NOQA
>>> max_index = 10
>>> subset_size = None
>>> seed = None
>>> rng = np.random.RandomState(0)
>>> subst = random_indexes(max_index, subset_size, seed, rng)
>>> result = ('subst = %s' % (str(subst),))
>>> print(result)
utool.util_numpy.random_sample(list_, nSample, strict=False, rng=<module 'numpy.random' from '/usr/local/lib/python2.7/dist-packages/numpy/random/__init__.pyc'>, seed=None)[source]

Grabs data randomly

Parameters:
  • list (list) –
  • nSample –
  • strict (bool) – (default = False)
  • rng (module) – random number generator(default = numpy.random)
  • seed (None) – (default = None)
Returns:

sample_list
Return type:

list

CommandLine:

python -m utool.util_numpy --exec-random_sample

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_numpy import *  # NOQA
>>> list_ = np.arange(10)
>>> nSample = 4
>>> strict = False
>>> rng = np.random.RandomState(0)
>>> seed = None
>>> sample_list = random_sample(list_, nSample, strict, rng, seed)
>>> result = ('sample_list = %s' % (str(sample_list),))
>>> print(result)
utool.util_numpy.sample_domain(min_, max_, nSamp, mode='linear')[source]

Example

>>> # ENABLE_DOCTEST
>>> import utool
>>> min_ = 10
>>> max_ = 1000
>>> nSamp  = 7
>>> result = utool.sample_domain(min_, max_, nSamp)
[10, 151, 293, 434, 576, 717, 859]
utool.util_numpy.spaced_indexes(len_, n, trunc=False)[source]

Returns n evenly spaced indexes. Returns as many as possible if trunc is true

utool.util_numpy.spaced_items(list_, n, **kwargs)[source]

Returns n evenly spaced items

utool.util_numpy.tiled_range(range_, cols)[source]

utool.util_parallel module

Module to executes the same function with different arguments in parallel.

class utool.util_parallel.KillableProcess(group=None, target=None, name=None, args=(), kwargs={})[source]

Bases: multiprocessing.process.Process

Simple subclass of multiprocessing.Process Gives an additional method to kill all children as well as itself. calls this function on delete.

terminate2()[source]
class utool.util_parallel.KillableThread(group=None, target=None, name=None, args=(), kwargs=None, verbose=None)[source]

Bases: threading.Thread

References

http://code.activestate.com/recipes/496960-thread2-killable-threads/ http://tomerfiliba.com/recipes/Thread2/

raise_exc(excobj)[source]
terminate()[source]
utool.util_parallel.buffered_generator(source_gen, buffer_size=2)[source]

Generator that runs a slow source generator in a separate process.

My generate function still seems faster on test cases. However, this function is more flexible in its compatability.

Parameters:
  • source_gen (iterable) – slow generator
  • buffer_size (int) – the maximal number of items to pre-generate (length of the buffer) (default = 2)
  • use_multiprocessing (bool) – if False uses GIL-hindered threading instead of multiprocessing (defualt = False).

Note

use_multiprocessing = True seems to freeze if passed in a generator built by six.moves.map.

References

Taken from Sander Dieleman’s data augmentation pipeline https://github.com/benanne/kaggle-ndsb/blob/11a66cdbddee16c69514b9530a727df0ac6e136f/buffering.py

CommandLine:

python -m utool.util_parallel --test-buffered_generator:0
python -m utool.util_parallel --test-buffered_generator:1

Example

>>> # UNSTABLE_DOCTEST
>>> from utool.util_parallel import *  # NOQA
>>> import utool as ut
>>> num = 2 ** 14
>>> func = ut.is_prime
>>> data = [38873] * num
>>> data = list(range(num))
>>> with ut.Timer('serial') as t1:
...     result1 = list(map(func, data))
>>> with ut.Timer('ut.generate') as t3:
...     result3 = list(ut.generate(func, data, chunksize=2, quiet=1, verbose=0))
>>> with ut.Timer('ut.buffered_generator') as t2:
...     result2 = list(ut.buffered_generator(map(func, data)))
>>> assert len(result1) == num and len(result2) == num and len(result3) == num
>>> assert result3 == result2, 'inconsistent results'
>>> assert result1 == result2, 'inconsistent results'
Example1:
>>> # VERYSLLOOWWW_DOCTEST
>>> from utool.util_parallel import _test_buffered_generator
>>> _test_buffered_generator2()
utool.util_parallel.close_pool(terminate=False, quiet=False)[source]
utool.util_parallel.ensure_pool(warn=False, quiet=False)[source]
utool.util_parallel.generate(func, args_list, ordered=True, force_serial=None, chunksize=None, prog=True, verbose=True, quiet=False, nTasks=None, freq=None, **kwargs)[source]

Provides an interfaces to python’s multiprocessing module. Esentially maps args_list onto func using pool.imap. Useful for embarrassingly parallel loops. Currently does not work with opencv3

Parameters:
  • func (function) – function to apply each argument to
  • args_list (list or iter) – sequence of tuples which are args for each function call
  • ordered (bool) –
  • force_serial (bool) –
  • chunksize (int) –
  • prog (bool) –
  • verbose (bool) –
  • nTasks (int) – optional (must be specified if args_list is an iterator)
Returns:

generator which yeilds result of applying func to args in args_list

CommandLine:

python -m utool.util_parallel --test-generate
python -m utool.util_parallel --test-generate:0
python -m utool.util_parallel --test-generate:0 --use-global-pool
python -m utool.util_parallel --test-generate:1
python -m utool.util_parallel --test-generate:2
python -m utool.util_parallel --test-generate:3
python -m utool.util_parallel --test-generate --verbose

python -c "import multiprocessing; print(multiprocessing.__version__)"
python -c "import cv2; print(cv2.__version__)"
Example0:
>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> #num = 8700  # parallel is slower for smaller numbers
>>> num = 500  # parallel has an initial (~.1 second startup overhead)
>>> print('TESTING SERIAL')
>>> flag_generator0 = ut.generate(ut.is_prime, range(0, num), force_serial=True, freq=num / 4)
>>> flag_list0 = list(flag_generator0)
>>> print('TESTING PARALLEL')
>>> flag_generator1 = ut.generate(ut.is_prime, range(0, num), freq=num / 10)
>>> flag_list1 = list(flag_generator1)
>>> print('ASSERTING')
>>> assert len(flag_list1) == num
>>> assert flag_list0 == flag_list1
Example1:
>>> # ENABLE_DOCTEST
>>> # Trying to recreate the freeze seen in IBEIS
>>> import utool as ut
>>> print('TESTING SERIAL')
>>> flag_generator0 = ut.generate(ut.is_prime, range(0, 1))
>>> flag_list0 = list(flag_generator0)
>>> flag_generator1 = ut.generate(ut.fibonacci_recursive, range(0, 1))
>>> flag_list1 = list(flag_generator1)
>>> print('TESTING PARALLEL')
>>> flag_generator2 = ut.generate(ut.is_prime, range(0, 12))
>>> flag_list2 = list(flag_generator2)
>>> flag_generator3 = ut.generate(ut.fibonacci_recursive, range(0, 12))
>>> flag_list3 = list(flag_generator3)
>>> print('flag_list0 = %r' % (flag_list0,))
>>> print('flag_list1 = %r' % (flag_list1,))
>>> print('flag_list2 = %r' % (flag_list1,))
>>> print('flag_list3 = %r' % (flag_list1,))
Example2:
>>> # UNSTABLE_DOCTEST
>>> # Trying to recreate the freeze seen in IBEIS
>>> import vtool as vt
>>> #def gen_chip(tup):
>>> #    import vtool as vt
>>> #    cfpath, gfpath, bbox, theta, new_size, filter_list = tup
>>> #    chipBGR = vt.compute_chip(gfpath, bbox, theta, new_size, filter_list)
>>> #    height, width = chipBGR.shape[0:2]
>>> #    vt.imwrite(cfpath, chipBGR)
>>> #    return cfpath, width, height
>>> import utool as ut
>>> from ibeis.algo.preproc.preproc_chip import gen_chip
>>> #from ibeis.algo.preproc.preproc_feat import gen_feat_worker
>>> key_list = ['grace.jpg', 'easy1.png', 'ada2.jpg', 'easy3.png',
>>>             'hard3.png', 'zebra.png', 'patsy.jpg', 'ada.jpg',
>>>             'carl.jpg', 'lena.png', 'easy2.png']
>>> img_fpath_list = [ut.grab_test_imgpath(key) for key in key_list]
>>> arg_list1 = [(ut.augpath(img_fpath, '_testgen'), img_fpath, (0, 0, 100, 100), 0.0, (545, 372), []) for img_fpath in img_fpath_list[0:1]]
>>> arg_list2 = [(ut.augpath(img_fpath, '_testgen'), img_fpath, (0, 0, 100, 100), 0.0, (545, 372), []) for img_fpath in img_fpath_list]
>>> #arg_list3 = [(count, fpath, {}) for count, fpath in enumerate(ut.get_list_column(arg_list1, 0))]
>>> #arg_list4 = [(count, fpath, {}) for count, fpath in enumerate(ut.get_list_column(arg_list2, 0))]
>>> ut.remove_file_list(ut.get_list_column(arg_list2, 0))
>>> chips1 = [x for x in ut.generate(gen_chip, arg_list1)]
>>> chips2 = [y for y in ut.generate(gen_chip, arg_list2, force_serial=True)]
>>> #feats3 = [z for z in ut.generate(gen_feat_worker, arg_list3)]
>>> #feats4 = [w for w in ut.generate(gen_feat_worker, arg_list4)]
Example3:
>>> # FAILING_DOCTEST
>>> # Trying to recreate the freeze seen in IBEIS
>>> # Extremely weird case: freezes only if dsize > (313, 313) AND __testwarp was called beforehand.
>>> # otherwise the parallel loop works fine. Could be an opencv 3.0.0-dev issue.
>>> import vtool as vt
>>> import utool as ut
>>> from ibeis.algo.preproc.preproc_chip import gen_chip
>>> import cv2
>>> from utool.util_parallel import __testwarp
>>> key_list = ['grace.jpg', 'easy1.png', 'ada2.jpg', 'easy3.png',
>>>             'hard3.png', 'zebra.png', 'patsy.jpg', 'ada.jpg',
>>>             'carl.jpg', 'lena.png', 'easy2.png']
>>> img_fpath_list = [ut.grab_test_imgpath(key) for key in key_list]
>>> arg_list1 = [(vt.imread(fpath),) for fpath in img_fpath_list[0:1]]
>>> arg_list2 = [(vt.imread(fpath),) for fpath in img_fpath_list]
>>> #new1 = [x for x in ut.generate(__testwarp, arg_list1)]
>>> new1 =  __testwarp(arg_list1[0])
>>> new2 = [y for y in ut.generate(__testwarp, arg_list2, force_serial=False)]
>>> #print('new2 = %r' % (new2,))

#Example4: # >>> # Freakin weird. When IBEIS Runs generate it doesn’t close the processes # >>> # UNSTABLE_DOCTEST # >>> # python -m utool.util_parallel –test-generate:4 # >>> # Trying to see if we can recreate the problem where there are # >>> # defunct python processes # >>> import utool as ut # >>> #num = 8700 # parallel is slower for smaller numbers # >>> num = 70000 # parallel has an initial (~.1 second startup overhead) # >>> print(‘TESTING PARALLEL’) # >>> flag_generator1 = list(ut.generate(ut.is_prime, range(0, num))) # >>> flag_generator1 = list(ut.generate(ut.is_prime, range(0, num))) # >>> import time # >>> time.sleep(10)

utool.util_parallel.get_default_numprocs()[source]
utool.util_parallel.in_main_process()[source]

Returns if you are executing in a multiprocessing subprocess Usefull to disable init print messages on windows

utool.util_parallel.init_pool(num_procs=None, maxtasksperchild=None, quiet=False, **kwargs)[source]

warning this might not be the right hting to do

utool.util_parallel.init_worker()[source]
utool.util_parallel.new_pool(num_procs, init_worker, maxtasksperchild)[source]
utool.util_parallel.process(func, args_list, args_dict={}, force_serial=None, nTasks=None, quiet=False)[source]

Use ut.generate rather than ut.process

Parameters:
  • func (func) –
  • args_list (list or iter) –
  • args_dict (dict) –
  • force_serial (bool) –
Returns:

result of parallel map(func, args_list)

CommandLine:

python -m utool.util_parallel --test-process

Example

>>> # SLOW_DOCTEST
>>> import utool as ut
>>> num = 8700  # parallel is slower for smaller numbers
>>> flag_generator0 = ut.process(ut.is_prime, list(zip(range(0, num))), force_serial=True)
>>> flag_list0 = list(flag_generator0)
>>> flag_generator1 = ut.process(ut.is_prime, list(zip(range(0, num))), force_serial=False)
>>> flag_list1 = list(flag_generator1)
>>> assert flag_list0 == flag_list1
utool.util_parallel.set_num_procs(num_procs)[source]
utool.util_parallel.spawn_background_daemon_thread(func, *args, **kwargs)[source]
utool.util_parallel.spawn_background_process(func, *args, **kwargs)[source]

Run a function in the background (like rebuilding some costly data structure)

References

http://stackoverflow.com/questions/2046603/is-it-possible-to-run-function-in-a-subprocess-without-threading-or-writing-a-se http://stackoverflow.com/questions/1196074/starting-a-background-process-in-python http://stackoverflow.com/questions/15063963/python-is-thread-still-running

Parameters:func (function) –

CommandLine:

python -m utool.util_parallel --test-spawn_background_process

Example

>>> # SLOW_DOCTEST
>>> from utool.util_parallel import *  # NOQA
>>> import utool as ut
>>> import time
>>> from os.path import join
>>> # build test data
>>> fname = 'test_bgfunc_output.txt'
>>> dpath = ut.get_app_resource_dir('utool')
>>> ut.ensuredir(dpath)
>>> fpath = join(dpath, fname)
>>> # ensure file is not around
>>> sleep_time = 1
>>> ut.delete(fpath)
>>> assert not ut.checkpath(fpath, verbose=True)
>>> def backgrond_func(fpath, sleep_time):
...     import utool as ut
...     import time
...     print('[BG] Background Process has started')
...     time.sleep(sleep_time)
...     print('[BG] Background Process is writing')
...     ut.write_to(fpath, 'background process')
...     print('[BG] Background Process has finished')
...     #raise AssertionError('test exception')
>>> # execute function
>>> func = backgrond_func
>>> args = (fpath, sleep_time)
>>> kwargs = {}
>>> print('[FG] Spawning process')
>>> threadid = ut.spawn_background_process(func, *args, **kwargs)
>>> assert threadid.is_alive() is True, 'thread should be active'
>>> print('[FG] Spawned process. threadid=%r' % (threadid,))
>>> # background process should not have finished yet
>>> assert not ut.checkpath(fpath, verbose=True)
>>> print('[FG] Waiting to check')
>>> time.sleep(sleep_time + .1)
>>> print('[FG] Finished waiting')
>>> # Now the file should be there
>>> assert ut.checkpath(fpath, verbose=True)
>>> assert threadid.is_alive() is False, 'process should have died'
utool.util_parallel.spawn_background_thread(func, *args, **kwargs)[source]

utool.util_path module

python -c “import utool, doctest; print(doctest.testmod(utool.util_path))”

This module becomes nav

class utool.util_path.ChdirContext(dpath=None, stay=False)[source]

Bases: object

References http://www.astropython.org/snippet/2009/10/chdir-context-manager

utool.util_path.append_suffixlist_to_namelist(name_list, suffix_list)[source]

adds a suffix to the path before the extension if name_list is a path_list the basepath is stripped away

utool.util_path.assert_exists(path_, msg=u'', **kwargs)

Asserts that a patha exists

utool.util_path.assertpath(path_, msg=u'', **kwargs)[source]

Asserts that a patha exists

utool.util_path.augpath(path, augsuf=u'', augext=u'', augpref=u'', augdir=None, newext=None, newfname=None, ensure=False)[source]

augments end of path before the extension.

augpath

Parameters:
  • path (str) –
  • augsuf (str) – augment filename before extension
Returns:

newpath
Return type:

str

Example

>>> from utool.util_path import *  # NOQA
>>> path = 'somefile.txt'
>>> augsuf = '_aug'
>>> newpath = augpath(path, augsuf)
>>> result = str(newpath)
>>> print(result)
somefile_aug.txt

Example

>>> from utool.util_path import *  # NOQA
>>> path = 'somefile.txt'
>>> augsuf = '_aug2'
>>> newext = '.bak'
>>> augdir = 'backup'
>>> newpath = augpath(path, augsuf, newext=newext, augdir=augdir)
>>> result = str(newpath)
>>> print(result)
backup/somefile_aug2.bak
utool.util_path.basename_noext(path)[source]
utool.util_path.checkpath(path_, verbose=False, n=None, info=False)[source]

verbose wrapper around os.path.exists

Returns:

true if path_ exists on the filesystem show only the top n directories
Parameters:
  • path (str) – path string
  • verbose (bool) – verbosity flag(default = False)
  • n (int) – (default = None)
  • info (bool) – (default = False)

CommandLine:

python -m utool.util_path --test-checkpath

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> path_ = ut.__file__
>>> verbose = True
>>> n = None
>>> info = False
>>> result = checkpath(path_, verbose, n, info)
>>> print(result)
True

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> path_ = ut.__file__ + 'foobar'
>>> verbose = True
>>> result = checkpath(path_, verbose, n=None, info=True)
>>> print(result)
False
utool.util_path.copy(src, dst, overwrite=True, deeplink=True, verbose=True, dryrun=False)[source]
utool.util_path.copy_all(src_dir, dest_dir, glob_str_list, recursive=False)[source]
utool.util_path.copy_files_to(src_fpath_list, dst_dpath=None, dst_fpath_list=None, overwrite=False, verbose=True, veryverbose=False)[source]

parallel copier

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *
>>> import utool as ut
>>> overwrite = False
>>> veryverbose = False
>>> verbose = True
>>> src_fpath_list = [ut.grab_test_imgpath(key)
>>>                   for key in ut.get_valid_test_imgkeys()]
>>> dst_dpath = ut.get_app_resource_dir('utool', 'filecopy_tests')
>>> copy_files_to(src_fpath_list, dst_dpath, overwrite=overwrite,
>>>               verbose=verbose)
utool.util_path.copy_list(src_list, dst_list, lbl=u'Copying', ioerr_ok=False, sherro_ok=False, oserror_ok=False)[source]

Copies all data and stat info

utool.util_path.copy_single(src, dst, overwrite=True, verbose=True, deeplink=True, dryrun=False)[source]
Parameters:
  • src (str) – file or directory to copy
  • dst (str) – directory or new file to copy to

Copies src file or folder to dst.

If src is a folder this copy is recursive.

utool.util_path.copy_worker(args)[source]

for util_parallel.generate

utool.util_path.delete(path, dryrun=False, recursive=True, verbose=False, print_exists=True, ignore_errors=True, **kwargs)[source]

Removes a file or directory

utool.util_path.dirsplit(path)[source]
Parameters:path (str) –
Returns:compoments of the path
Return type:list

CommandLine:

python -m utool.util_path --exec-dirsplit

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> paths = []
>>> paths.append('E:/window file/foo')
>>> paths.append('/normal/foo')
>>> paths.append('~/relative/path')
>>> results = [dirsplit(path) for path in paths]
>>> import re
>>> results2 = [re.split('\\/', path) for path in paths]
>>> print(results2)
>>> result = ut.repr2(results)
>>> print(result)
utool.util_path.ensure_crossplat_path(path, winroot=u'C:')[source]
Parameters:path (str) –
Returns:crossplat_path
Return type:str
Example(DOCTEST):
>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> path = r'C:\somedir'
>>> cplat_path = ensure_crossplat_path(path)
>>> result = cplat_path
>>> print(result)
C:/somedir
utool.util_path.ensure_ext(fname, ext)[source]
Parameters:
  • fname (str) – file name
  • ext (str) –
Returns:

fname - file name
Return type:

str

CommandLine:

python -m utool.util_path --exec-ensure_ext --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> print(ensure_ext('foo', '.bar'))
>>> print(ensure_ext('foo.bar', '.bar'))
utool.util_path.ensure_mingw_drive(win32_path)[source]

replaces windows drives with mingw style drives

Parameters:win32_path (str) –

CommandLine:

python -m utool.util_path --test-ensure_mingw_drive

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> win32_path = r'C:/Program Files/Foobar'
>>> result = ensure_mingw_drive(win32_path)
>>> print(result)
/c/Program Files/Foobar
utool.util_path.ensure_native_path(path, winroot=u'C:')[source]
utool.util_path.ensure_unixslash(path)[source]
utool.util_path.ensuredir(path_, verbose=False, info=False, mode=1023)[source]

Ensures that directory will exist. creates new dir with sticky bits by default

utool.util_path.ensurepath(path_, verbose=False)[source]

DEPRICATE - alias - use ensuredir instead

utool.util_path.existing_commonprefix(paths)[source]
utool.util_path.existing_subpath(root_path, valid_subpaths, tiebreaker=u'first', verbose=False)[source]

Returns join(root_path, subpath) where subpath in valid_subpath ane exists(subpath)

utool.util_path.expand_win32_shortname(path1)[source]
utool.util_path.file_bytes(fpath)[source]
Parameters:fpath (str) – file path string
Returns:size of file in bytes
Return type:int
utool.util_path.file_megabytes(fpath)[source]
Parameters:fpath (str) – file path string
Returns:size of file in megabytes
Return type:float
utool.util_path.find_lib_fpath(libname, root_dir, recurse_down=True, verbose=False, debug=False)[source]

Search for the library

utool.util_path.fnames_to_fpaths(fname_list, path)[source]
utool.util_path.fpaths_to_fnames(fpath_list)[source]
Parameters:fpath_list (list of strs) – list of file-paths
Returns:fname_list – list of file-names
Return type:list of strs
utool.util_path.get_basename_noext_list(path_list)[source]
utool.util_path.get_basepath_list(path_list)[source]
utool.util_path.get_ext_list(path_list)[source]
utool.util_path.get_modname_from_modpath(module_fpath)[source]

returns importable name from file path

get_modname_from_modpath

Parameters:module_fpath (str) – module filepath
Returns:modname
Return type:str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> module_fpath = ut.util_path.__file__
>>> modname = ut.get_modname_from_modpath(module_fpath)
>>> result = modname
>>> print(result)
utool.util_path
utool.util_path.get_modpath_from_modname(modname, prefer_pkg=False, prefer_main=False)[source]
Parameters:modname (str) –
Returns:module_dir
Return type:str

CommandLine:

python -m utool.util_path --test-get_modpath_from_modname
Setup:
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> utool_dir = dirname(dirname(ut.__file__))

Example

>>> # ENABLE_DOCTEST
>>> modname = 'utool.util_path'
>>> module_dir = get_modpath_from_modname(modname)
>>> result = ut.truepath_relative(module_dir, utool_dir)
>>> result = ut.ensure_unixslash(result)
>>> print(result)
utool/util_path.py

Example

>>> # ENABLE_DOCTEST
>>> modname = 'utool._internal'
>>> module_dir = get_modpath_from_modname(modname, prefer_pkg=True)
>>> result = ut.truepath_relative(module_dir, utool_dir)
>>> result = ut.ensure_unixslash(result)
>>> print(result)
utool/_internal

Example

>>> # ENABLE_DOCTEST
>>> modname = 'utool'
>>> module_dir = get_modpath_from_modname(modname)
>>> result = ut.truepath_relative(module_dir, utool_dir)
>>> result = ut.ensure_unixslash(result)
>>> print(result)
utool/__init__.py
utool.util_path.get_module_dir(module, *args)[source]
utool.util_path.get_module_subdir_list(module_fpath)[source]
Parameters:module_fpath (str) –

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> module_fpath = ut.util_path.__file__
>>> modsubdir_list = get_module_subdir_list(module_fpath)
>>> result = modsubdir_list
>>> print(result)
['utool', 'util_path']
utool.util_path.get_path_type(path_)[source]

returns if a path is a file, directory, link, or mount

utool.util_path.get_relative_modpath(module_fpath)[source]

Returns path to module relative to the package root

Parameters:module_fpath (str) – module filepath
Returns:modname
Return type:str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> module_fpath = ut.util_path.__file__
>>> rel_modpath = ut.get_relative_modpath(module_fpath)
>>> rel_modpath = rel_modpath.replace('.pyc', '.py')  # allow pyc or py
>>> result = ensure_crossplat_path(rel_modpath)
>>> print(result)
utool/util_path.py
utool.util_path.get_standard_exclude_dnames()[source]
utool.util_path.get_standard_include_patterns()[source]
utool.util_path.get_win32_short_path_name(long_name)[source]

Gets the short path name of a given long path.

References

http://stackoverflow.com/a/23598461/200291 http://stackoverflow.com/questions/23598289/get-win-short-fname-python

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut  # NOQA
>>> # build test data
>>> #long_name = unicode(normpath(ut.get_resource_dir()))
>>> long_name = unicode(r'C:/Program Files (x86)')
>>> #long_name = unicode(r'C:/Python27')
#unicode(normpath(ut.get_resource_dir()))
>>> # execute function
>>> result = get_win32_short_path_name(long_name)
>>> # verify results
>>> print(result)
C:/PROGRA~2
utool.util_path.glob(dpath, pattern=None, recursive=False, with_files=True, with_dirs=True, maxdepth=None, exclude_dirs=[], fullpath=True, **kwargs)[source]

Globs directory for pattern

Parameters:
  • dpath (str) – directory path or pattern
  • pattern (str or list) – pattern or list of patterns (use only if pattern is not in dpath)
  • recursive (bool) – (default = False)
  • with_files (bool) – (default = True)
  • with_dirs (bool) – (default = True)
  • maxdepth (None) – (default = None)
  • exclude_dirs (list) – (default = [])
Returns:

path_list
Return type:

list
SeeAlso:
iglob

CommandLine:

python -m utool.util_path --test-glob
python -m utool.util_path --exec-glob:1

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> from os.path import dirname
>>> dpath = dirname(ut.__file__)
>>> pattern = '__*.py'
>>> recursive = True
>>> with_files = True
>>> with_dirs = True
>>> maxdepth = None
>>> fullpath = False
>>> exclude_dirs = ['_internal', join(dpath, 'experimental')]
>>> print('exclude_dirs = ' + ut.list_str(exclude_dirs))
>>> path_list = glob(dpath, pattern, recursive, with_files, with_dirs,
>>>                  maxdepth, exclude_dirs, fullpath)
>>> result = ('path_list = %s' % (ut.list_str(path_list),))
>>> result = result.replace(r'\\', '/')
>>> print(result)
path_list = [
    '__init__.py',
    '__main__.py',
    'tests/__init__.py',
]

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> dpath = dirname(ut.__file__) + '/__*.py'
>>> path_list = glob(dpath)
>>> result = ('path_list = %s' % (str(path_list),))
>>> print(result)
utool.util_path.glob_python_modules(dirname, **kwargs)[source]
utool.util_path.grep(regex_list, recursive=True, dpath_list=None, include_patterns=None, exclude_dirs=[], greater_exclude_dirs=None, inverse=False, verbose=False, fpath_list=None, reflags=0, cache=None)[source]

greps for patterns Python implementation of grep. NOT FINISHED

Parameters:
  • regex_list (str or list) – one or more patterns to find
  • recursive (bool) –
  • dpath_list (list) – directories to search (defaults to cwd)
  • include_patterns (list) – defaults to standard file extensions
Returns:

(found_fpaths, found_lines_list, found_lxs_list)
Return type:

(list, list, list)

CommandLine:

python -m utool.util_path --test-grep
utprof.py -m utool.util_path --exec-grep
utprof.py utool/util_path.py --exec-grep

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> #dpath_list = [ut.truepath('~/code/utool/utool')]
>>> dpath_list = [ut.truepath(dirname(ut.__file__))]
>>> include_patterns = ['*.py']
>>> exclude_dirs = []
>>> regex_list = ['grepfile']
>>> verbose = True
>>> recursive = True
>>> result = ut.grep(regex_list, recursive, dpath_list, include_patterns,
>>>                  exclude_dirs)
>>> (found_fpath_list, found_lines_list, found_lxs_list) = result
>>> assert 'util_path.py' in list(map(basename, found_fpath_list))
utool.util_path.grepfile(fpath, regexpr_list, reflags=0, cache=None)[source]

grepfile - greps a specific file

Parameters:
  • fpath (str) –
  • regexpr_list (list or str) – pattern or list of patterns
Returns:

tuple – list of lines and list of line numbers
Return type:

list, list

CommandLine:

python -m utool.util_path --exec-grepfile

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> fpath = ut.get_modpath_from_modname(ut.util_path)
>>> regexpr_list = ['foundthisline', '__future__']
>>> cache = None
>>> reflags = 0
>>> found_lines, found_lxs = ut.grepfile(fpath, regexpr_list)
>>> result = ut.repr3({'found_lines': found_lines, 'found_lxs': found_lxs})
>>> print(result)
>>> assert 7 in found_lxs
>>> others = ut.take_complement(found_lxs, [found_lxs.index(7)])
>>> assert others[0] == others[1]
utool.util_path.iglob(dpath, pattern=None, recursive=False, with_files=True, with_dirs=True, maxdepth=None, exclude_dirs=[], fullpath=True, **kwargs)[source]

Iteratively globs directory for pattern

FIXME:
This function has a speed issue
Parameters:
  • dpath (str) – directory path
  • pattern (str) –
  • recursive (bool) – (default = False)
  • with_files (bool) – (default = True)
  • with_dirs (bool) – (default = True)
  • maxdepth (None) – (default = None)
  • exclude_dirs (list) – (default = [])
Yields:

path

References

http://stackoverflow.com/questions/19859840/excluding-dirs-in-os-walk

utool.util_path.is_module_dir(path)[source]
Parameters:path (str) –
Returns:True if path contains an __init__ file
Return type:flag

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> path = truepath('~/code/utool/utool')
>>> flag = is_module_dir(path)
>>> result = (flag)
>>> print(result)
utool.util_path.is_private_module(path)[source]
utool.util_path.is_python_module(path)[source]
utool.util_path.list_images(img_dpath_, ignore_list=[], recursive=False, fullpath=False, full=None, sort=True)[source]

Returns a list of images in a directory. By default returns relative paths.

TODO: rename to ls_images TODO: Change all instances of fullpath to full

Parameters:
  • img_dpath (str) –
  • ignore_list (list) – (default = [])
  • recursive (bool) – (default = False)
  • fullpath (bool) – (default = False)
  • full (None) – (default = None)
  • sort (bool) – (default = True)
Returns:

gname_list
Return type:

list

CommandLine:

python -m utool.util_path --exec-list_images

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> img_dpath_ = '?'
>>> ignore_list = []
>>> recursive = False
>>> fullpath = False
>>> full = None
>>> sort = True
>>> gname_list = list_images(img_dpath_, ignore_list, recursive,
>>>                          fullpath, full, sort)
>>> result = ('gname_list = %s' % (str(gname_list),))
>>> print(result)
utool.util_path.longest_existing_path(_path)[source]

Returns the longest root of _path that exists

Parameters:_path (str) – path string
Returns:_path - path string
Return type:str

CommandLine:

python -m utool.util_path --exec-longest_existing_path

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> target = dirname(ut.__file__)
>>> _path = join(target, 'nonexist/foobar')
>>> existing_path = longest_existing_path(_path)
>>> result = ('existing_path = %s' % (str(existing_path),))
>>> print(result)
>>> assert existing_path == target
utool.util_path.ls(path, pattern=u'*')[source]

like unix ls - lists all files and dirs in path

utool.util_path.ls_dirs(path, pattern=u'*')[source]
utool.util_path.ls_images(img_dpath_, ignore_list=[], recursive=False, fullpath=False, full=None, sort=True)

Returns a list of images in a directory. By default returns relative paths.

TODO: rename to ls_images TODO: Change all instances of fullpath to full

Parameters:
  • img_dpath (str) –
  • ignore_list (list) – (default = [])
  • recursive (bool) – (default = False)
  • fullpath (bool) – (default = False)
  • full (None) – (default = None)
  • sort (bool) – (default = True)
Returns:

gname_list
Return type:

list

CommandLine:

python -m utool.util_path --exec-list_images

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> img_dpath_ = '?'
>>> ignore_list = []
>>> recursive = False
>>> fullpath = False
>>> full = None
>>> sort = True
>>> gname_list = list_images(img_dpath_, ignore_list, recursive,
>>>                          fullpath, full, sort)
>>> result = ('gname_list = %s' % (str(gname_list),))
>>> print(result)
utool.util_path.ls_moduledirs(path, private=True, full=True)[source]

lists all dirs which are python modules in path

utool.util_path.ls_modulefiles(path, private=True, full=True, noext=False)[source]
utool.util_path.make_grep_resultstr(grep_result, extended_regex_list, reflags, colored=True)[source]
utool.util_path.matches_image(fname)[source]

returns true if a filename matches an image pattern

utool.util_path.matching_fnames(dpath_list, include_patterns, exclude_dirs=[], greater_exclude_dirs=[], recursive=True)[source]

matching_fnames. walks dpath lists returning all directories that match the requested pattern.

Parameters:
  • dpath_list (list) –
  • include_patterns (str) –
  • exclude_dirs (None) –
  • recursive (bool) –

References

# TODO: fix names and behavior of exclude_dirs and greater_exclude_dirs http://stackoverflow.com/questions/19859840/excluding-directories-in-os-walk

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> dpath_list = [dirname(dirname(ut.__file__))]
>>> include_patterns = get_standard_include_patterns()
>>> exclude_dirs = ['_page']
>>> greater_exclude_dirs = get_standard_exclude_dnames()
>>> recursive = True
>>> fpath_gen = matching_fnames(dpath_list, include_patterns, exclude_dirs,
>>>                             greater_exclude_dirs, recursive)
>>> result = list(fpath_gen)
>>> print('\n'.join(result))
utool.util_path.move(src, dst, lbl=u'Moving')[source]
utool.util_path.move_list(src_list, dst_list, lbl=u'Moving')[source]
utool.util_path.newcd(path)[source]

DEPRICATE

utool.util_path.num_images_in_dir(path)[source]

returns the number of images in a directory

utool.util_path.path_ndir_split(path_, n, force_unix=True, winroot=u'C:', trailing=True)[source]

Shows only a little bit of the path. Up to the n bottom-level directories

TODO: rename to path_tail? ndir_split?

Returns:(str) the trailing n paths of path.

CommandLine:

python3 -m utool.util_path --test-path_ndir_split
python3 -m utool --tf path_ndir_split
python -m utool --tf path_ndir_split

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> paths = [r'/usr/bin/local/foo/bar',
...          r'C:/',
...          #r'lonerel',
...          #r'reldir/other',
...          r'/ham',
...          r'./eggs',
...          r'/spam/eggs',
...          r'C:\Program Files (x86)/foobar/bin',]
>>> N = 2
>>> iter_ = ut.iprod(paths, range(1, N + 1))
>>> force_unix = True
>>> tuplist = [(n, path_ndir_split(path_, n)) for path_, n in iter_]
>>> chunklist = list(ut.ichunks(tuplist, N))
>>> list_ = [['n=%r: %s' % (x, ut.reprfunc(y)) for x, y in chunk]
>>>          for chunk in chunklist]
>>> line_list = [', '.join(strs) for strs in list_]
>>> result = '\n'.join(line_list)
>>> print(result)
n=1: '.../bar', n=2: '.../foo/bar'
n=1: 'C:/', n=2: 'C:/'
n=1: '.../ham', n=2: '/ham'
n=1: '.../eggs', n=2: './eggs'
n=1: '.../eggs', n=2: '.../spam/eggs'
n=1: '.../bin', n=2: '.../foobar/bin'
utool.util_path.pathsplit_full(path)[source]

splits all directories in path into a list

utool.util_path.platform_path(path)[source]

Returns platform specific path for pyinstaller usage

Parameters:path (str) –
Returns:path2
Return type:str

CommandLine:

python -m utool.util_path --test-platform_path

Example

>>> # ENABLE_DOCTEST
>>> # FIXME: find examples of the wird paths this fixes (mostly on win32 i think)
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> path = 'some/odd/../weird/path'
>>> path2 = platform_path(path)
>>> result = str(path2)
>>> if ut.WIN32:
...     ut.assert_eq(path2, r'some\weird\path')
... else:
...     ut.assert_eq(path2, r'some/weird/path')

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut    # NOQA
>>> if ut.WIN32:
...     path = 'C:/PROGRA~2'
...     path2 = platform_path(path)
...     assert path2 == u'..\\..\\..\\..\\Program Files (x86)'
utool.util_path.relpath_unix(path, otherpath)[source]
utool.util_path.remove_dirs(dpath, dryrun=False, ignore_errors=True, quiet=False, **kwargs)[source]

Recursively removes a single directory (need to change function name)

DEPRICATE

Parameters:
  • dpath (str) – directory path
  • dryrun (bool) – (default = False)
  • ignore_errors (bool) – (default = True)
  • quiet (bool) – (default = False)
Returns:

False
Return type:

bool

CommandLine:

python -m utool.util_path --test-remove_dirs

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> dpath = ut.ensure_app_resource_dir('utool', 'testremovedir')
>>> assert exists(dpath), 'nothing to remove'
>>> dryrun = False
>>> ignore_errors = True
>>> quiet = False
>>> flag = remove_dirs(dpath, dryrun, ignore_errors, quiet)
>>> result = ('flag = %s' % (flag,))
>>> print(result)
>>> assert not exists(dpath), 'did not remove dpath'
flag = True
utool.util_path.remove_existing_fpaths(fpath_list, verbose=False, quiet=False, strict=False, print_caller=False, lbl=u'files')[source]

checks existance before removing. then tries to remove exisint paths

utool.util_path.remove_file(fpath, verbose=True, dryrun=False, ignore_errors=True, **kwargs)[source]

Removes a file

utool.util_path.remove_file_list(fpath_list, verbose=False, quiet=False, strict=False, print_caller=False, lbl=u'files')

Removes multiple file paths

utool.util_path.remove_files_in_dir(dpath, fname_pattern_list=u'*', recursive=False, verbose=False, dryrun=False, ignore_errors=False, **kwargs)[source]

Removes files matching a pattern from a directory

utool.util_path.remove_fpaths(fpath_list, verbose=False, quiet=False, strict=False, print_caller=False, lbl=u'files')[source]

Removes multiple file paths

utool.util_path.search_candidate_paths(candidate_path_list, candidate_name_list=None, priority_paths=None, required_subpaths=[], verbose=True)[source]

searches for existing paths that meed a requirement

Parameters:
  • candidate_path_list (list) – list of paths to check. If candidate_name_list is specified this is the dpath list instead
  • candidate_name_list (list) – specifies several names to check (default = None)
  • priority_paths (None) – specifies paths to check first. Ignore candidate_name_list (default = None)
  • required_subpaths (list) – specified required directory structure (default = [])
  • verbose (bool) – verbosity flag(default = True)
Returns:

return_path
Return type:

str

CommandLine:

python -m utool.util_path --test-search_candidate_paths

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> candidate_path_list = [ut.truepath('~/RPI/code/utool'),
>>>                        ut.truepath('~/code/utool')]
>>> candidate_name_list = None
>>> required_subpaths = []
>>> verbose = True
>>> priority_paths = None
>>> return_path = search_candidate_paths(candidate_path_list,
>>>                                      candidate_name_list,
>>>                                      priority_paths, required_subpaths,
>>>                                      verbose)
>>> result = ('return_path = %s' % (str(return_path),))
>>> print(result)
utool.util_path.search_in_dirs(fname, search_dpaths=[], shortcircuit=True, return_tried=False, strict=False)[source]
Parameters:
  • fname (str) – file name
  • search_dpaths (list) –
  • shortcircuit (bool) –
  • return_tried (bool) – return tried paths
  • strict (bool) – (default = False)
Returns:

None
Return type:

fpath

Example

>>> import utool as ut
>>> fname = 'Inno Setup 5\ISCC.exe'
>>> search_dpaths = ut.get_install_dirs()
>>> shortcircuit = True
>>> fpath = ut.search_in_dirs(fname, search_dpaths, shortcircuit)
>>> print(fpath)
utool.util_path.sed(regexpr, repl, force=False, recursive=False, dpath_list=None, fpath_list=None, verbose=None)[source]

Python implementation of sed. NOT FINISHED

searches and replaces text in files

Parameters:
  • regexpr (str) – regx patterns to find
  • repl (str) – text to replace
  • force (bool) –
  • recursive (bool) –
  • dpath_list (list) – directories to search (defaults to cwd)
utool.util_path.sedfile(fpath, regexpr, repl, force=False, verbose=True, veryverbose=False)[source]

Executes sed on a specific file

Parameters:
  • fpath (str) – file path string
  • regexpr (str) –
  • repl (str) –
  • force (bool) – (default = False)
  • verbose (bool) – verbosity flag(default = True)
  • veryverbose (bool) – (default = False)
Returns:

changed_lines
Return type:

list

CommandLine:

python -m utool.util_path --exec-sedfile --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> fpath = ut.get_modpath_from_modname(ut.util_path)
>>> regexpr = 'sedfile'
>>> repl = 'saidfile'
>>> force = False
>>> verbose = True
>>> veryverbose = False
>>> changed_lines = sedfile(fpath, regexpr, repl, force, verbose, veryverbose)
>>> result = ('changed_lines = %s' % (ut.repr3(changed_lines),))
>>> print(result)

Attempt to create unix or windows symlink TODO: TEST / FIXME

Parameters:
  • path (str) – path to real file or directory
  • link (str) – path to desired location for symlink
  • noraise (bool) –
utool.util_path.tail(fpath, n=2, trailing=True)[source]

Alias for path_ndir_split

utool.util_path.testgrep()[source]

utprof.py -m utool.util_path –exec-testgrep

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> #dpath_list = [ut.truepath('~/code/utool/utool')]
>>> dpath_list = [ut.truepath(dirname(ut.__file__))]
>>> include_patterns = ['*.py']
>>> exclude_dirs = []
>>> regex_list = ['grepfile']
>>> verbose = True
>>> recursive = True
>>> result = ut.grep(regex_list, recursive, dpath_list, include_patterns,
>>>                  exclude_dirs)
>>> (found_fpath_list, found_lines_list, found_lxs_list) = result
>>> assert 'util_path.py' in list(map(basename, found_fpath_list))
utool.util_path.touch(fname, times=None, verbose=True)[source]
Parameters:
  • fname (str) –
  • times (None) –
  • verbose (bool) –

Example

>>> from utool.util_path import *  # NOQA
>>> fname = '?'
>>> times = None
>>> verbose = True
>>> result = touch(fname, times, verbose)
>>> print(result)

References

http://stackoverflow.com/questions/1158076/implement-touch-using-python

utool.util_path.truepath_relative(path, otherpath=None)[source]

Normalizes and returns absolute path with so specs

Parameters:
  • path (str) – path to file or directory
  • otherpath (None) – (default = None)
Returns:

path_
Return type:

str

CommandLine:

python -m utool.util_path --exec-truepath_relative --show

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_path import *  # NOQA
>>> import utool as ut
>>> path = 'C:/foobar/foobiz'
>>> otherpath = 'C:/foobar'
>>> path_ = truepath_relative(path, otherpath)
>>> result = ('path_ = %s' % (ut.repr2(path_),))
>>> print(result)
path_ = 'foobiz'
utool.util_path.unexpanduser(path)[source]
utool.util_path.win_shortcut(source, link_name)[source]

Attempt to create windows shortcut TODO: TEST / FIXME

References

http://stackoverflow.com/questions/1447575/symlinks-on-windows

utool.util_print module

class utool.util_print.Indenter(lbl=' ', enabled=True)[source]

Bases: object

Monkey patches modules injected with print to change the way print behaves.

Works with utool.inject to allow for prefixing of all within-context print statements in a semi-dynamic manner. There seem to be some bugs but it works pretty well.

CommandLine:

python -m utool.util_print --exec-Indenter

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_print import *  # NOQA
>>> import utool as ut
>>> ut.util_print._test_indent_print()
start()[source]
stop()[source]
utool.util_print.colorprint(text, color=None)[source]

provides some color to terminal output

Parameters:
  • text (str) –
  • color (str) –

CommandLine:

python -c "import pygments.console; print(list(pygments.console.codes.keys()))"
python -m utool.util_print --exec-colorprint
python -m utool.util_print --exec-colorprint:1

import pygments
print(ut.list_str(list(pygments.formatters.get_all_formatters())))
print(list(pygments.styles.get_all_styles()))
Example0:
>>> # DISABLE_DOCTEST
>>> from utool.util_print import *  # NOQA
>>> import pygments.console
>>> msg_list = list(pygments.console.codes.keys())
>>> color_list = list(pygments.console.codes.keys())
>>> [colorprint(text, color) for text, color in zip(msg_list, color_list)]
Example1:
>>> # DISABLE_DOCTEST (Windows test)
>>> from utool.util_print import *  # NOQA
>>> import pygments.console
>>> print('line1')
>>> colorprint('line2', 'red')
>>> colorprint('line3', 'blue')
>>> colorprint('line4', 'fuchsia')
>>> colorprint('line5', 'reset')
>>> colorprint('line5', 'fuchsia')
>>> print('line6')
utool.util_print.dictprint(dict_, dict_name=None, **kwargs)
utool.util_print.horiz_print(*args)[source]
utool.util_print.printNOTQUIET(msg)[source]
utool.util_print.printVERBOSE(msg, verbarg)[source]
utool.util_print.printWARN(msg)[source]
utool.util_print.print_code(text, lexer_name='python')[source]
Parameters:text (str) –

CommandLine:

python -m utool.util_print --test-print_python_code

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_print import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> text = ut.read_from(ut.__file__.replace('.pyc', '.py'))
>>> # execute function
>>> print_python_code(text)
utool.util_print.print_dict(dict_, dict_name=None, **kwargs)[source]
utool.util_print.print_difftext(text, other=None)[source]
Parameters:text (str) –

CommandLine:

#python -m utool.util_print --test-print_difftext
#autopep8 ingest_data.py --diff | python -m utool.util_print --test-print_difftext
utool.util_print.print_filesize(fpath)[source]
utool.util_print.print_list(list_, **kwargs)[source]
utool.util_print.print_locals()[source]
utool.util_print.print_python_code(text)[source]
SeeAlso:
print_code
utool.util_print.printdict(dict_, dict_name=None, **kwargs)
utool.util_print.printif(func, condition=False)[source]

execute printfunc only if condition=QUIET

utool.util_print.printshape(arr_name, locals_)[source]

utool.util_progress module

progress handler.

Old progress funcs needto be depricated ProgressIter and ProgChunks are pretty much the only useful things here.

utool.util_progress.PROGRESS_FLUSH()

flush() -> None. Flush the internal I/O buffer.

utool.util_progress.ProgChunks(list_, chunksize, nInput=None, **kwargs)[source]

Yeilds an iterator in chunks and computes progress Progress version of ut.ichunks

class utool.util_progress.ProgIter(iterable, lbl=u'Prog', adjust=True, freq=1, **kwargs)[source]

Bases: utool.util_progress.ProgressIter

utool.util_progress.ProgPartial(*args, **kwargs)[source]
class utool.util_progress.ProgressIter(iterable=None, *args, **kwargs)[source]

Bases: object

Wraps a for loop with progress reporting

lbl=’Progress: ‘, nTotal=0, flushfreq=4, startafter=-1, start=True, repl=False, approx=False, disable=False, writefreq=1, with_time=False, backspace=True, pad_stdout=False, wfreq=None, ffreq=None, freq=None, total=None, num=None, with_totaltime=None

Referencs:
https://github.com/verigak/progress/blob/master/progress/__init__.py
Parameters:
  • () (iterable) – iterable normally passed to for loop
  • lbl (str) – progress label
  • nTotal (int) –
  • flushfreq (int) –
  • startafter (int) –
  • start (bool) –
  • repl (bool) –
  • approx (bool) –
  • enabled (bool) –
  • writefreq (int) –
  • with_totaltime (bool) –
  • backspace (bool) –
  • pad_stdout (bool) –
  • autoadjust (bool) – no adjusting frequency if True (default False)
  • wfreq (None) – alias for write_freq
  • ffreq (None) – alias for flush_freq
  • total (None) – alias for nTotal
  • num (None) – alias for nTotal
Timeit::

import utool as ut setup = ut.codeblock( ‘’’ import utool as ut from six.moves import range, zip import time def time_append(size):

start_time = time.time() last_time = start_time list2 = [] for x in range(size):

now_time = time.time() between = now_time - last_time last_time = now_time list2.append(between)
def time_assign(size):

start_time = time.time() last_time = start_time list1 = ut.alloc_nones(size) for x in range(size):

now_time = time.time() between = now_time - last_time last_time = now_time list1[x] = between
def time_baseline(size):

start_time = time.time() last_time = start_time for x in range(size):

now_time = time.time() between = now_time - last_time last_time = now_time
def time_null(size):
for x in range(size):
pass

‘’‘)

input_sizes = [2 ** count for count in range(7, 12)] stmt_list = [‘time_assign’, ‘time_append’, ‘time_baseline’, ‘time_null’] input_sizes=[100, 1000, 10000] ut.timeit_grid(stmt_list, setup, input_sizes=input_sizes, show=True)

CommandLine:

python -m utool.util_progress --test-ProgressIter
python -m utool.util_progress --test-ProgressIter:0
python -m utool.util_progress --test-ProgressIter:1
python -m utool.util_progress --test-ProgressIter:2

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> from six.moves import range
>>> num = 1000
>>> num2 = 10001
>>> results1 = [x for x in ut.ProgressIter(range(num), wfreq=10, adjust=True)]
>>> results4 = [x for x in ut.ProgressIter(range(num), wfreq=1, adjust=True)]
>>> results2 = [x for x in range(num)]
>>> results3 = [x for x in ut.progiter((y + 1 for y in range(num2)),
>>>                                    nTotal=num2, wfreq=1000,
>>>                                    backspace=True, adjust=True)]
>>> assert results1 == results2
Example1:
>>> # SLOW_DOCTEST
>>> import utool as ut
>>> from six.moves import range
>>> num2 = 10001
>>> progiter = ut.ProgressIter(range(num2), lbl='testing primes',
>>>                            report_unit='seconds', freq=1,
>>>                            time_thresh=.1, adjust=True)
>>> results1 = [ut.get_nth_prime_bruteforce(29) for x in progiter]
>>> print(ut.truncate_str(ut.repr2(results1)))
Example2:
>>> # SLOW_DOCTEST
>>> import utool as ut
>>> from six.moves import range
>>> num2 = 100001
>>> progiter = ut.ProgressIter(range(num2), lbl='testing primes',
>>>                            report_unit='seconds', freq=1,
>>>                            time_thresh=3, adjust=True)
>>> results1 = [ut.get_nth_prime_bruteforce(29) for x in progiter]
>>> print(ut.truncate_str(ut.repr2(results1)))
static build_msg_fmtstr2(lbl, nTotal, invert_rate, backspace)[source]
Parameters:
  • lbl (str) –
  • invert_rate (bool) –
  • backspace (bool) –
Returns:

msg_fmtstr_time
Return type:

str

CommandLine:

python -m utool.util_progress --exec-ProgressIter.build_msg_fmtstr2
Setup:
>>> from utool.util_progress import *  # NOQA
>>> lbl = 'foo'
>>> invert_rate = True
>>> backspace = False
>>> nTotal = None

Example

>>> # DISABLE_DOCTEST
>>> msg_fmtstr_time = ProgressIter.build_msg_fmtstr2(lbl, nTotal, invert_rate, backspace)
>>> result = ('%s' % (ut.repr2(msg_fmtstr_time),))
>>> print(result)
static build_msg_fmtstr_head_cols(nTotal, lbl)[source]
get_subindexers(prog_iter, num_substeps)[source]
iter_rate()[source]

pun not intended

# TODO: record iteration times for analysis # TODO Incorporate this better # FIXME; pad_stdout into subfunctions

import dis dis.dis(ut.ProgressIter.iter_rate)

iter_without_rate()[source]
mark_current()[source]
utool.util_progress.get_nTotalChunks(nTotal, chunksize)[source]

Returns the number of chunks that a list will be split into given a chunksize.

Parameters:
  • nTotal (int) –
  • chunksize (int) –
Returns:

nTotalChunks
Return type:

int
SeeAlso:
util_iter.ichunks

CommandLine:

python -m utool.util_progress --exec-get_nTotalChunks

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_progress import *  # NOQA
>>> nTotal = 2000
>>> chunksize = 256
>>> nTotalChunks = get_nTotalChunks(nTotal, chunksize)
>>> result = ('nTotalChunks = %s' % (six.text_type(nTotalChunks),))
>>> print(result)
nTotalChunks = 8
utool.util_progress.log_progress(lbl=u'Progress: ', nTotal=0, flushfreq=4, startafter=-1, start=True, repl=False, approx=False, disable=False, writefreq=1, with_time=False, backspace=True, pad_stdout=False, wfreq=None, ffreq=None, freq=None, total=None, num=None, with_totaltime=None)[source]

FIXME: depricate for ProgressIter. Still used in smk funcs

Returns two functions (mark_progress, end_progress) which will handle logging progress in a for loop.

flush frequency must be a multiple of write frequency

Parameters:
  • lbl (str) – progress label
  • nTotal (int) –
  • flushfreq (int) –
  • startafter (int) –
  • start (bool) –
  • repl (bool) –
  • approx (bool) –
  • disable (bool) –
  • writefreq (int) –
  • with_totaltime (bool) –
  • backspace (bool) –
  • pad_stdout (bool) –
  • wfreq (None) – alias for write_freq
  • ffreq (None) – alias for flush_freq
  • freq (None) – alias for flush_freq and write_freq (prefered)
  • total (None) – alias for nTotal
  • num (None) – alias for nTotal

Example

>>> import utool, time
>>> from six.moves import range
>>> # Define a dummy task
>>> spam = 42.0
>>> nTotal = 1000
>>> iter_ = (num for num in range(0, nTotal * 2, 2))
>>> # Create progress functions
... mark_, end_ = utool.log_progress('prog ', nTotal, flushfreq=17)
prog    0/1000
>>> for count, item in enumerate(iter_):  
...     # Call with enumerate to keep track of a count variable
...     time.sleep(.001)
...     spam += item + count
...     mark_(count)
...prog 1000/1000
>>> # Mark completion
>>> end_()
utool.util_progress.progiter

alias of ProgressIter

utool.util_progress.progress_str(max_val, lbl=u'Progress: ', repl=False, approx=False, backspace=True)[source]

makes format string that prints progress: %Xd/MAX_VAL with backspaces

NOTE: r can be used instead of backspaces. This function is not very relevant because of that.

utool.util_progress.simple_progres_func(verbosity, msg, progchar=u'.')[source]
utool.util_progress.test_progress()[source]

CommandLine:

python -m utool.util_progress --test-test_progress

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_progress import *  # NOQA
>>> test_progress()

utool.util_project module

class utool.util_project.GrepResult(found_fpath_list, found_lines_list, found_lxs_list, extended_regex_list, reflags)[source]

Bases: utool.util_dev.NiceRepr

hack_remove_pystuff()[source]
make_resultstr(colored=True)[source]
remove_results(indicies)[source]
class utool.util_project.UserProfile[source]

Bases: object

utool.util_project.ensure_text(fname, text, repo_dpath='.', force=False, locals_={}, chmod=None)[source]
Parameters:
  • fname (str) – file name
  • text (str) –
  • repo_dpath (str) – directory path string(default = ‘.’)
  • force (bool) – (default = False)
  • locals (dict) – (default = {})

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_git import *  # NOQA
>>> import utool as ut
>>> result = setup_repo()
>>> print(result)
utool.util_project.ensure_user_profile(user_profile=None)[source]
Parameters:user_profile (UserProfile) – (default = None)
Returns:user_profile
Return type:UserProfile

CommandLine:

python -m utool.util_project --exec-ensure_user_profile --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_project import *  # NOQA
>>> import utool as ut
>>> user_profile = None
>>> user_profile = ensure_user_profile(user_profile)
>>> result = ('user_profile = %s' % (ut.repr2(user_profile),))
>>> print(ut.repr3(user_profile.project_dpaths))
>>> print(result)
utool.util_project.grep_projects(tofind_list, user_profile=None, verbose=True, new=False, **kwargs)[source]

Greps the projects defined in the current UserProfile

Parameters:
  • tofind_list (list) –
  • user_profile (None) – (default = None)
Kwargs:
user_profile

CommandLine:

python -m utool --tf grep_projects grep_projects

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_project import *  # NOQA
>>> import utool as ut
>>> import sys
>>> tofind_list = ut.get_argval('--find', type_=list,
>>>                             default=[sys.argv[-1]])
>>> grep_projects(tofind_list)
utool.util_project.ibeis_user_profile()[source]
utool.util_project.sed_projects(regexpr, repl, force=False, recursive=True, user_profile=None, **kwargs)[source]
Parameters:
  • regexpr –
  • repl –
  • force (bool) – (default = False)
  • recursive (bool) – (default = True)
  • user_profile (None) – (default = None)

CommandLine:

python -m utool.util_project --exec-sed_projects

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_project import *  # NOQA
>>> regexpr = ut.get_argval('--find', type_=str, default=sys.argv[-1])
>>> repl = ut.get_argval('--repl', type_=str, default=sys.argv[-2])
>>> force = False
>>> recursive = True
>>> user_profile = None
>>> result = sed_projects(regexpr, repl, force, recursive, user_profile)
>>> print(result)
utool.util_project.setup_repo()[source]

Creates default structure for a new repo

CommandLine:

python -m utool --tf setup_repo --repo=dtool --codedir=~/code

python -m utool --tf setup_repo --repo=dtool --codedir=~/code
python -m utool --tf setup_repo --repo=ibeis-flukematch-module --codedir=~/code --modname=ibeis_flukematch
python -m utool --tf setup_repo --repo=mtgmonte --codedir=~/code --modname=mtgmonte
python -m utool --tf setup_repo --repo=pydarknet --codedir=~/code --modname=pydarknet

python -m utool --tf setup_repo
Python:
ipython import utool as ut ut.rrrr(0); ut.setup_repo()

Example

>>> # SCRIPT
>>> from utool.util_git import *  # NOQA
>>> import utool as ut
>>> result = setup_repo()
>>> print(result)

utool.util_regex module

in vim nongreedy .* is .{-} in python nongreedy .* is .*?

utool.util_regex.backref_field(key)[source]
utool.util_regex.bref_field(key)
utool.util_regex.convert_text_to_varname(text)[source]
Parameters:text (str) – text that might not be a valid variablename
Returns:varname
Return type:str

CommandLine:

python -m utool.util_regex --test-convert_text_to_varname

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_regex import *  # NOQA
>>> text = '0) View Application-Files Directory. '
>>> varname = convert_text_to_varname(text)
>>> result = ('varname = %s' % (str(varname),))
>>> print(result)
_0_View_ApplicationFiles_Directory_
utool.util_regex.extend_regex(regexpr)[source]
Extends the syntax of regular expressions by replacing convineince syntax
with re friendly syntax. Nameely things that I use in vim like <>
utool.util_regex.extend_regex2(regexpr, reflags=0)[source]

also preprocesses flags

utool.util_regex.extend_regex3(regex_list, reflags=0)[source]
utool.util_regex.get_match_text(match)[source]
utool.util_regex.modify_quoted_strs(text, modify_func=None)[source]

doesnt work with escaped quotes or multilines single quotes only. no nesting.

Parameters:
  • text –
  • modify_func (None) –

Example

>>> from utool.util_regex import *  # NOQA
>>> text = "'just' 'a' sentance with 'strings' in it "
>>> text2 = "'this' 'text' wont work 'because 'of "the"' "nesting"'"
>>> text3 = " ''' god "help" you ''' if you use 'triple quotes'  "
>>> def modify_func(quoted_str):
...     return quoted_str.upper()
>>> result = modify_quoted_strs(text, modify_func)
>>> print(result)
'JUST' 'A' sentance with 'STRINGS'
utool.util_regex.named_field(key, regex, vim=False)[source]

Creates a named regex group that can be referend via a backref. If key is None the backref is referenced by number.

References

https://docs.python.org/2/library/re.html#regular-expression-syntax

utool.util_regex.named_field_regex(keypat_tups)[source]
Parameters:
  • keypat_tups (list) – tuples of (name, pattern) or a string for an unnamed
  • pattern –
Returns:

regex
Return type:

str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_regex import *  # NOQA
>>> keypat_tups = [
...    ('name',  r'G\d+'),  # species and 2 numbers
...    ('under', r'_'),     # 2 more numbers
...    ('id',    r'\d+'),   # 2 more numbers
...    ( None,   r'\.'),
...    ('ext',   r'\w+'),
... ]
>>> regex = named_field_regex(keypat_tups)
>>> result = (regex)
>>> print(result)
(?P<name>G\d+)(?P<under>_)(?P<id>\d+)(\.)(?P<ext>\w+)
utool.util_regex.named_field_repl(field_list)[source]
Parameters:field_list (list) – list of either a tuples to denote a keyword, or a strings for relacement t3ext
Returns:repl for regex
Return type:str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_regex import *  # NOQA
>>> field_list = [('key',), 'unspecial string']
>>> repl = named_field_repl(field_list)
>>> result = repl
>>> print(result)
\g<key>unspecial string
utool.util_regex.negative_lookahead(regex, vim=False)[source]
utool.util_regex.negative_lookbehind(regex, vim=False)[source]
Parameters:regex –
Returns:
Return type:?

CommandLine:

python -m utool.util_regex --exec-negative_lookbehind

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_regex import *  # NOQA
>>> regex = re.escape('\'')
>>> pattern = negative_lookbehind(regex) + 'foo'
>>> match1 = re.search(pattern, '\'foo\'')
>>> match2 = re.search(pattern, '"foo"')
>>> match3 = re.search(pattern, '\' foo\'')
>>> match4 = re.search(pattern, '" foo"')
utool.util_regex.padded_parse(pattern, text)[source]
utool.util_regex.parse_docblock(func_code)[source]

#TODO: Finish me

References

http://pyparsing.wikispaces.com/share/view/1264103 http://code.activestate.com/recipes/576704-python-code-minifier/

Example

>>> import utool as ut
>>> import inspect
>>> func_code = inspect.getsource(ut.modify_quoted_strs)
>>> func_code =
utool.util_regex.parse_python_syntax(text)[source]

step1: split lines

step2: parse enclosure pairity for each line to find unended lines

for each unending line, is there a valid merge line? (a line that could snytatically finish this runnon statement? If no then error. Else try to join the two lines.

step3: perform context_sensitive_edit

Example

>>> from utool.util_regex import *   # NOQA
>>> import utool
>>> from os.path import normpath
>>> text = utool.read_from(utool.util_regex.__file__)
utool.util_regex.positive_lookahead(regex, vim=False)[source]
utool.util_regex.positive_lookbehind(regex, vim=False)[source]
utool.util_regex.regex_get_match(regex, text, fromstart=False)[source]
utool.util_regex.regex_matches(regex, text, fromstart=True)[source]
utool.util_regex.regex_or(list_)[source]
utool.util_regex.regex_parse(regex, text, fromstart=True)[source]
Parameters:
  • regex (str) –
  • text (str) –
  • fromstart (bool) –
Returns:
Return type:

dict or None

Example

>>> from utool.util_regex import *  # NOQA
>>> regex = r'(?P<string>\'[^\']*\')'
>>> text = " 'just' 'a' sentance with 'strings' in it "
>>> fromstart = False
>>> result = regex_parse(regex, text, fromstart)['string']
>>> print(result)
utool.util_regex.regex_replace(regex, repl, text)[source]

thin wrapper around re.sub regex_replace

MULTILINE and DOTALL are on by default in all util_regex functions

Parameters:
  • regex (str) – pattern to find
  • repl (str) – replace pattern with this
  • text (str) – text to modify
Returns:

modified text
Return type:

str
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_regex import *  # NOQA
>>> regex = r'\(.*\):'
>>> repl = '(*args)'
>>> text = '''def foo(param1,
...                   param2,
...                   param3):'''
>>> result = regex_replace(regex, repl, text)
>>> print(result)
def foo(*args)
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_regex import *  # NOQA
>>> import utool as ut
>>> regex = ut.named_field_regex([('keyword', 'def'), ' ', ('funcname', '.*'), '\(.*\):'])
>>> repl = ut.named_field_repl([('funcname',), ('keyword',)])
>>> text = '''def foo(param1,
...                   param2,
...                   param3):'''
>>> result = regex_replace(regex, repl, text)
>>> print(result)
foodef
utool.util_regex.regex_replace_lines(lines, regexpat, replpat)[source]
utool.util_regex.regex_split(regex, text)[source]
utool.util_regex.regex_word(w)[source]
utool.util_regex.whole_word(regex)[source]

utool.util_resources module

utool.util_resources.available_memory()[source]

Returns total system wide available memory in bytes

utool.util_resources.current_memory_usage()[source]

Returns this programs current memory usage in bytes

utool.util_resources.get_matching_process_ids(cmd_pattern, user_pattern)[source]

CommandLine:

export PID=30196
export PID=$(python -c "import utool as ut; print(ut.get_matching_process_ids('jonc', 'python2.7'))")
export PID=$(python -c "import utool as ut; print(ut.get_matching_process_ids('jonc', 'matlab'))")
sudo -H echo $PID
ps -o pid,comm,nice -p $PID
renice 10 -p $PID
sudo renice -4 -p $PID

user_pattern = ‘jonc’ cmd_pattern = ‘main.py’ user_pattern = None cmd_pattern = ‘matlab’ get_matching_process_ids(cmd_pattern, user_pattern)

utool.util_resources.get_memstats_str()[source]
utool.util_resources.get_python_datastructure_sizes()[source]

References

http://stackoverflow.com/questions/1331471/in-memory-size-of-python-stucture

CommandLine:

python -m utool.util_resources --test-get_python_datastructure_sizes

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_resources import *  # NOQA
>>> import utool as ut  # NOQA
>>> type_sizes = get_python_datastructure_sizes()
>>> result = ut.dict_str(type_sizes, sorted_=True)
>>> print(result)
utool.util_resources.get_resource_limits()[source]
utool.util_resources.get_resource_usage_str()[source]
utool.util_resources.memstats()[source]
utool.util_resources.num_cpus()[source]
utool.util_resources.peak_memory()[source]

Returns the resident set size (the portion of a process’s memory that is held in RAM.)

utool.util_resources.print_resource_usage()[source]
utool.util_resources.time_in_systemmode()[source]
utool.util_resources.time_in_usermode()[source]
utool.util_resources.time_str2(seconds)[source]
utool.util_resources.total_memory()[source]

Returns total system wide memory in bytes

utool.util_resources.used_memory()[source]

Returns total system wide used memory in bytes

utool.util_set module

class utool.util_set.OrderedSet(iterable=None)[source]

Bases: _abcoll.MutableSet

Set the remembers the order elements were added
Big-O running times for all methods are the same as for regular sets. The internal self.__map dictionary maps keys to links in a doubly linked list. The circular doubly linked list starts and ends with a sentinel element. The sentinel element never gets deleted (this simplifies the algorithm). The prev/next links are weakref proxies (to prevent circular references). Individual links are kept alive by the hard reference in self.__map. Those hard references disappear when a key is deleted from an OrderedSet.

References

http://code.activestate.com/recipes/576696/ http://code.activestate.com/recipes/576694/ http://stackoverflow.com/questions/1653970/does-python-have-an-ordered-set

add(key)[source]
append(key)[source]
discard(key)[source]
pop(last=True)[source]
utool.util_set.oset

alias of OrderedSet

utool.util_setup module

utool.util_setup.NOOP()[source]
class utool.util_setup.SETUP_PATTERNS[source]
chmod_test = ['test_*.py']
clutter_cyth = ['_*_cyth.o', '_*_cyth_bench.py', 'run_cyth_benchmarks.sh', '_*_cyth.c', '_*_cyth.pyd', '_*_cyth.pxd', '_*_cyth.html', '_*_cyth.pyx', '_*_cyth.so', '_*_cyth.dylib']
clutter_pybuild = ['*.pyc', '*.pyo']
utool.util_setup.assert_in_setup_repo(setup_fpath, name='')[source]

pass in __file__ from setup.py

utool.util_setup.autogen_sphinx_apidoc()[source]

autogen_sphinx_docs.py

CommandLine:

python -m utool.util_setup --exec-autogen_sphinx_apidoc

Example

>>> # SCRIPT
>>> from utool.util_setup import *  # NOQA
>>> autogen_sphinx_apidoc()
utool.util_setup.build_pyo(project_dirs)[source]
utool.util_setup.clean(setup_dir, clutter_patterns, clutter_dirs)[source]
utool.util_setup.find_ext_modules(disable_warnings=True)[source]
utool.util_setup.find_packages(recursive=True, maxdepth=None)[source]

Finds all directories with an __init__.py file in them

utool.util_setup.get_cmdclass()[source]
utool.util_setup.get_numpy_include_dir()[source]
utool.util_setup.parse_author()[source]

TODO: this function should parse setup.py or a module for the author variable

utool.util_setup.parse_package_for_version(name)[source]

Searches for a variable named __version__ in name’s __init__.py file and returns the value. This function parses the source text. It does not load the module.

utool.util_setup.parse_readme(readmefile='README.md')[source]
utool.util_setup.presetup(setup_fpath, kwargs)
utool.util_setup.presetup_commands(setup_fpath, kwargs)[source]
utool.util_setup.read_license(license_file)[source]
utool.util_setup.setup_chmod(setup_fpath, setup_dir, chmod_patterns)[source]

Gives files matching pattern the same chmod flags as setup.py

utool.util_setup.setuptools_setup(setup_fpath=None, module=None, **kwargs)[source]

Arguments which can be passed to setuptools:

============       =====            ===========
Install-Data       Value            Description
------------       -----            -----------
*packages          strlist          a list of packages modules to be distributed
py_modules         strlist          a list of singlefile modules to be distributed
scripts            strlist          a list of standalone scripts to build and install
*install_requires  list             e.g: ['distribute == 0.7.3', 'numpy', 'matplotlib']
data_files         strlist          a list of data files to install
zip_safe           bool             install efficiently installed as a zipped module?
namespace_packages list             packages without meaningful __init__.py's
package_dir        dict             keys are packagenames ('' is the root)
package_data       dict             keys are foldernames, values are a list of globstrs
*entry_pionts      dict             installs a script {'console_scripts': ['script_name_to_install = entry_module:entry_function']}

============       =====            ===========
Meta-Data          Value            Description
------------       -----            -----------
name               short string     ('name of the package')
version            short string     ('version of this release')
author             short string     ('package authors name')
author_email       email address    ('email address of the package author')
maintainer         short string     ('package maintainers name')
maintainer_email   email address    ('email address of the package maintainer')
url                URL              ('home page for the package')
description        short string     ('short, summary description of the package')
long_description   long string      ('longer description of the package')
download_url       URL              ('location where the package may be downloaded')
classifiers        list of strings  ('a list of classifiers')
platforms          list of strings  ('a list of platforms')
license            short string     ('license for the package')
utool.util_setup.translate_cyth()[source]

utool.util_six module

utool.util_sqlite module

class utool.util_sqlite.SQLColumnRichInfo(column_id, name, type_, notnull, dflt_value, pk)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__getstate__()

Exclude the OrderedDict from pickling

__repr__()

Return a nicely formatted representation string

column_id

Alias for field number 0

dflt_value

Alias for field number 4

name

Alias for field number 1

notnull

Alias for field number 3

pk

Alias for field number 5

type_

Alias for field number 2

utool.util_sqlite.get_nonprimary_columninfo(cur, tablename)[source]
utool.util_sqlite.get_primary_columninfo(cur, tablename)[source]
utool.util_sqlite.get_table_column(cur, tablename, colname)[source]

Conveinience:

utool.util_sqlite.get_table_columninfo_list(cur, tablename)[source]
Parameters:tablename (str) – table name
Returns:list of tuples with format: (
[0] column_id : id of the column [1] name : the name of the column [2] type_ : the type of the column (TEXT, INT, etc...) [3] notnull : 0 or 1 if the column can contains null values [4] dflt_value : the default value [5] pk : 0 or 1 if the column partecipate to the primary key

)
Return type:column_list

References

http://stackoverflow.com/questions/17717829/how-to-get-column-names-from-a-table-in-sqlite-via-pragma-net-c

CommandLine:

python -m utool.util_sqlite --test-get_table_columninfo_list

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_sqlite import *  # NOQA
utool.util_sqlite.get_table_columnname_list(cur, tablename)[source]
utool.util_sqlite.get_table_columns(cur, tablename, exclude_columns=[])[source]
utool.util_sqlite.get_table_csv(cur, tablename, exclude_columns=[])[source]

Conveinience: Converts a tablename to csv format

Parameters:
  • tablename (str) –
  • exclude_columns (list) –
Returns:

csv_table
Return type:

str

CommandLine:

python -m ibeis.control.SQLDatabaseControl --test-get_table_csv

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.control.SQLDatabaseControl import *  # NOQA
>>> # build test data
>>> import ibeis
>>> ibs = ibeis.opendb('testdb1')
>>> db = ibs.db
>>> tablename = ibeis.const.NAME_TABLE
>>> exclude_columns = []
>>> # execute function
>>> csv_table = db.get_table_csv(tablename, exclude_columns)
>>> # verify results
>>> result = str(csv_table)
>>> print(result)
utool.util_sqlite.get_table_num_rows(cur, tablename)[source]
utool.util_sqlite.get_table_rows(cur, tablename, colnames, where=None, params=None, unpack=True)[source]
utool.util_sqlite.get_tablenames(cur)[source]

Conveinience:

utool.util_sqlite.print_database_structure(cur)[source]

utool.util_str module

Module that handles string formating and manipulation of varoius data

utool.util_str.align(text, character=u'=', replchar=None, pos=0)[source]

Left justifies text on the left side of character

align

Parameters:
  • text (str) – text to align
  • character (str) – character to align at
  • replchar (str) – replacement character (default=None)
Returns:

new_text
Return type:

str

CommandLine:

python -m utool.util_str --test-align:0
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> character = '='
>>> text = 'a = b=\none = two\nthree = fish\n'
>>> print(text)
>>> result = (align(text, '='))
>>> print(result)
a     = b=
one   = two
three = fish
utool.util_str.align_lines(line_list, character=u'=', replchar=None, pos=0)[source]

Left justifies text on the left side of character

align_lines

Parameters:
  • line_list (list of strs) –
  • character (str) –
Returns:

new_lines
Return type:

list

CommandLine:

python -m utool.util_str --test-align_lines:0
python -m utool.util_str --test-align_lines:1
python -m utool.util_str --test-align_lines:2
python -m utool.util_str --test-align_lines:3
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> line_list = 'a = b\none = two\nthree = fish'.split('\n')
>>> character = '='
>>> new_lines = align_lines(line_list, character)
>>> result = ('\n'.join(new_lines))
>>> print(result)
a     = b
one   = two
three = fish
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> line_list = 'foofish:\n    a = b\n    one    = two\n    three    = fish'.split('\n')
>>> character = '='
>>> new_lines = align_lines(line_list, character)
>>> result = ('\n'.join(new_lines))
>>> print(result)
foofish:
    a        = b
    one      = two
    three    = fish
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> character = ':'
>>> text = ut.codeblock('''
    {'max': '1970/01/01 02:30:13',
     'mean': '1970/01/01 01:10:15',
     'min': '1970/01/01 00:01:41',
     'range': '2:28:32',
     'std': '1:13:57',}''').split('\n')
>>> new_lines = align_lines(text, ':', ' :')
>>> result = '\n'.join(new_lines)
>>> print(result)
{'max'   : '1970/01/01 02:30:13',
 'mean'  : '1970/01/01 01:10:15',
 'min'   : '1970/01/01 00:01:41',
 'range' : '2:28:32',
 'std'   : '1:13:57',}
Example3:
>>> # ENABLE_DOCEST
>>> from utool.util_str import *  # NOQA
>>> line_list = 'foofish:\n a = b = c\n one = two = three\nthree=4= fish'.split('\n')
>>> character = '='
>>> # align the second occurence of a character
>>> new_lines = align_lines(line_list, character, pos=None)
>>> print(('\n'.join(line_list)))
>>> result = ('\n'.join(new_lines))
>>> print(result)
foofish:
 a   = b   = c
 one = two = three
three=4    = fish
utool.util_str.array2string2(a, max_line_width=None, precision=None, suppress_small=None, separator=u' ', prefix=u'', style=<built-in function repr>, formatter=None, threshold=None)[source]

expanded version of np.core.arrayprint.array2string

utool.util_str.array_repr2(arr, max_line_width=None, precision=None, suppress_small=None, force_dtype=False, with_dtype=None, **kwargs)[source]

extended version of np.core.numeric.array_repr

ut.editfile(np.core.numeric.__file__)

On linux: _typelessdata [numpy.int64, numpy.float64, numpy.complex128, numpy.int64]

On BakerStreet _typelessdata [numpy.int32, numpy.float64, numpy.complex128, numpy.int32]

# WEIRD np.int64 is np.int64 _typelessdata[0] is _typelessdata[-1] _typelessdata[0] == _typelessdata[-1]

TODO:
replace force_dtype with with_dtype

id(_typelessdata[-1]) id(_typelessdata[0])

from numpy.core.numeric import _typelessdata _typelessdata

References

http://stackoverflow.com/questions/28455982/why-are-there-two-np-int64s -in-numpy-core-numeric-typelessdata-why-is-numpy-in/28461928#28461928

utool.util_str.autoformat_pep8(sourcecode, **kwargs)[source]
Parameters:code (str) –

CommandLine:

python -m utool.util_str --exec-autoformat_pep8
Kwargs:
‘aggressive’: 0, ‘diff’: False, ‘exclude’: [], ‘experimental’: False, ‘files’: [u’‘], ‘global_config’: ~/.config/pep8, ‘ignore’: set([u’E24’]), ‘ignore_local_config’: False, ‘in_place’: False, ‘indent_size’: 4, ‘jobs’: 1, ‘line_range’: None, ‘list_fixes’: False, ‘max_line_length’: 79, ‘pep8_passes’: -1, ‘recursive’: False, ‘select’: , ‘verbose’: 0,
utool.util_str.bbox_str(bbox, pad=4, sep=u', ')[source]

makes a string from an integer bounding box

utool.util_str.bubbletext(text, font=u'cybermedium')[source]

Other fonts include: cybersmall, cybermedium, and cyberlarge

import pyfiglet TODO move elsewhere

References

http://www.figlet.org/

Example

>>> # ENABLE_DOCTEST
>>> import utool as ut
>>> bubble_text1 = ut.bubbletext('TESTING', font='cyberlarge')
>>> bubble_text2 = ut.bubbletext('BUBBLE', font='cybermedium')
>>> bubble_text3 = ut.bubbletext('TEXT', font='cyberlarge')
>>> print('\n'.join([bubble_text1, bubble_text2, bubble_text3]))
utool.util_str.byte_str(nBytes, unit=u'bytes', precision=2)[source]

representing the number of bytes with the chosen unit

Returns:str
utool.util_str.byte_str2(nBytes, precision=2)[source]

Automatically chooses relevant unit (KB, MB, or GB) for displaying some number of bytes.

Parameters:nBytes (int) –
Returns:
Return type:str

CommandLine:

python -m utool.util_str --exec-byte_str2

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> nBytes_list = [1, 100, 1024,  1048576, 1073741824, 1099511627776]
>>> result = ut.list_str(list(map(byte_str2, nBytes_list)), nl=False)
>>> print(result)
['0.00 KB', '0.10 KB', '1.00 KB', '1.00 MB', '1.00 GB', '1.00 TB']
utool.util_str.chr_range(*args, **kw)[source]

Like range but returns characters

Parameters:
  • start (None) – (default = None)
  • stop (None) – (default = None)
  • step (None) – (default = None)
Returns:
Return type:

list

CommandLine:

python -m ibeis.algo.hots.bayes --exec-chr_range

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> args = (5,)
>>> result = ut.repr2(chr_range(2, base='a'))
>>> print(result)
>>> print(chr_range(0, 5))
>>> print(chr_range(0, 50))
>>> print(chr_range(0, 5, 2))
['a', 'b']
utool.util_str.clipstr(str_, maxlen)[source]

tries to shorten string as much as it can until it is just barely readable

utool.util_str.closet_words(query, options, num=1)[source]
utool.util_str.codeblock(block_str)[source]

Convinience function for defining code strings. Esspecially useful for templated code.

utool.util_str.color_text(text, color)[source]
SeeAlso:
highlight_text
utool.util_str.conj_phrase(list_, cond=u'or')[source]

Joins a list of words using English conjunction rules

Parameters:
  • list (list) – of strings
  • cond (str) – a conjunction (or, and, but)
Returns:

the joined cconjunction phrase
Return type:

str

References

http://en.wikipedia.org/wiki/Conjunction_(grammar)

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> list_ = ['a', 'b', 'c']
>>> result = conj_phrase(list_, 'or')
>>> print(result)
a, b, or c
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> list_ = ['a', 'b']
>>> result = conj_phrase(list_, 'and')
>>> print(result)
a and b
utool.util_str.countdown_flag(count_or_bool)[source]
utool.util_str.dict_itemstr_list(dict_, strvals=False, sorted_=None, newlines=True, recursive=True, indent_=u'', precision=None, hack_liststr=False, explicit=False, truncate=False, key_order=None, truncatekw={}, key_order_metric=None, use_numpy=True, with_comma=True, **dictkw)[source]
Returns:a list of human-readable dictionary items
Return type:list
Parameters:explicit – if True uses dict(key=val,...) format instead of {key:val,...}
utool.util_str.dict_str(dict_, strvals=False, sorted_=None, newlines=True, recursive=True, indent_=u'', precision=None, hack_liststr=None, truncate=False, nl=None, explicit=False, truncatekw={}, key_order=None, key_order_metric=None, nobraces=False, nobr=None, align=False, **dictkw)[source]
Makes a pretty printable / human-readable string representation of a
dictionary. In most cases this string could be evaled.
Parameters:
  • dict (dict_) – a dictionary
  • strvals (bool) – (default = False)
  • sorted (None) – returns str sorted by a metric (default = None)
  • newlines (bool) – Use nl instead. (default = True)
  • recursive (bool) – (default = True)
  • indent (str) – (default = ‘’)
  • precision (int) – (default = 8)
  • hack_liststr (bool) – turn recursive liststr parsing on (default = False)
  • truncate (bool) – (default = False)
  • nl (int) – prefered alias for newline. can be a coundown variable (default = None)
  • explicit (bool) – (default = False)
  • truncatekw (dict) – (default = {})
  • key_order (None) – overrides default ordering (default = None)
  • key_order_metric (str) – special sorting of items. Accepted values: None, ‘strlen’, ‘val’
  • nobraces (bool) – (default = False)
  • align (bool) – (default = False)
Kwargs:
use_numpy, with_comma
Returns:
Return type:str
FIXME:
ALL LIST DICT STRINGS ARE VERY SPAGEHETTI RIGHT NOW

CommandLine:

python -m utool.util_str --test-dict_str
python -m utool.util_str --test-dict_str --truncate=False --no-checkwant
python -m utool.util_str --test-dict_str --truncate=1 --no-checkwant
python -m utool.util_str --test-dict_str --truncate=2 --no-checkwant

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import dict_str, dict_itemstr_list
>>> import utool as ut
>>> dict_ = {'foo': {'spam': 'barbarbarbarbar' * 3, 'eggs': 'jam'},
>>>          'baz': 'barbarbarbarbar' * 3}
>>> truncate = ut.get_argval('--truncate', type_=None, default=1)
>>> result = dict_str(dict_, strvals=True, truncate=truncate,
>>>                    truncatekw={'maxlen': 20})
>>> print(result)
{
    'baz': barbarbarbarbarbarbarbarbarbarbarbarbarbarbar,
    'foo': {
        'eggs': jam,
        's ~~~TRUNCATED~~~ ,
    },
}
utool.util_str.difftext(text1, text2, num_context_lines=0, ignore_whitespace=False)

Uses difflib to return a difference string between two similar texts

References

http://www.java2s.com/Code/Python/Utility/IntelligentdiffbetweentextfilesTimPeters.htm

Parameters:
  • text1 –
  • text2 –
Returns:
Return type:

?

CommandLine:

python -m utool.util_str --test-get_textdiff:1
python -m utool.util_str --test-get_textdiff:0

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> # build test data
>>> text1 = 'one\ntwo\nthree'
>>> text2 = 'one\ntwo\nfive'
>>> # execute function
>>> result = get_textdiff(text1, text2)
>>> # verify results
>>> print(result)
- three
+ five
Example2:
>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> # build test data
>>> text1 = 'one\ntwo\nthree\n3.1\n3.14\n3.1415\npi\n3.4\n3.5\n4'
>>> text2 = 'one\ntwo\nfive\n3.1\n3.14\n3.1415\npi\n3.4\n4'
>>> # execute function
>>> num_context_lines = 1
>>> result = get_textdiff(text1, text2, num_context_lines)
>>> # verify results
>>> print(result)
utool.util_str.doctest_code_line(line_str, varname=None, verbose=True)[source]
utool.util_str.doctest_repr(var, varname=None, precision=2, verbose=True)[source]
utool.util_str.edit_distance(query, options)[source]

pip install python-Levenshtein

query = ‘hello world’ options = [‘goodbye world’, ‘rofl’, ‘hello’, ‘world’, ‘lowo’] [7, 9, 6, 6, 7]

utool.util_str.ensure_ascii(str_)[source]
utool.util_str.ensure_unicode_strlist(str_list)[source]
utool.util_str.file_megabytes_str(fpath)[source]
utool.util_str.filesize_str(fpath)[source]
utool.util_str.filtered_infostr(flags, lbl, reason=None)[source]
utool.util_str.find_block_end(row, line_list, sentinal, direction=1)[source]

Searches up and down until it finds the endpoints of a block Rectify with find_paragraph_end in pyvim_funcs

utool.util_str.flatten_textlines(text)[source]
utool.util_str.format_multi_paragraphs(text, debug=False, **kwargs)

FIXME: funky things happen when multiple newlines in the middle of paragraphs

CommandLine:

python ~/local/vim/rc/pyvim_funcs.py --test-format_multiple_paragraph_sentences

CommandLine:

python -m utool.util_str --exec-format_multiple_paragraph_sentences --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import os, sys
>>> #sys.path.append(os.path.expanduser('~/local/vim/rc'))
>>> text = testdata_text(2)
>>> formated_text = format_multiple_paragraph_sentences(text, debug=True)
>>> print('+--- Text ---')
>>> print(text)
>>> print('+--- Formated Text ---')
>>> print(formated_text)
>>> print('L_____')
utool.util_str.format_multiple_paragraph_sentences(text, debug=False, **kwargs)[source]

FIXME: funky things happen when multiple newlines in the middle of paragraphs

CommandLine:

python ~/local/vim/rc/pyvim_funcs.py --test-format_multiple_paragraph_sentences

CommandLine:

python -m utool.util_str --exec-format_multiple_paragraph_sentences --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import os, sys
>>> #sys.path.append(os.path.expanduser('~/local/vim/rc'))
>>> text = testdata_text(2)
>>> formated_text = format_multiple_paragraph_sentences(text, debug=True)
>>> print('+--- Text ---')
>>> print(text)
>>> print('+--- Formated Text ---')
>>> print(formated_text)
>>> print('L_____')
utool.util_str.format_single_paragraph_sentences(text, debug=False, myprefix=True, sentence_break=True)[source]

helps me separatate sentences grouped in paragraphs that I have a difficult time reading due to dyslexia

Parameters:text (str) –
Returns:wrapped_text
Return type:str

CommandLine:

python -m utool.util_str --exec-format_single_paragraph_sentences --show
python -m utool.util_str --exec-format_single_paragraph_sentences --show --nobreak

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> text = '     lorium ipsum doloar dolar dolar dolar erata man foobar is this there yet almost man not quit ate 80 chars yet hold out almost there? dolar erat. sau.ltum. fds.fd... . . fd oob fd. list: (1) abcd, (2) foobar (4) 123456789 123456789 123456789 123456789 123 123 123 123 123456789 123 123 123 123 123456789 123456789 123456789 123456789 123456789 123 123 123 123 123 123456789 123456789 123456789 123456789 123456789 123456789 (3) spam.'
>>> #text = 'list: (1) abcd, (2) foobar (3) spam.'
>>> #text = 'foo. when: (1) there is a new individual,'
>>> #text = 'when: (1) there is a new individual,'
>>> #text = '? ? . lorium. ipsum? dolar erat. saultum. fds.fd...  fd oob fd. ? '  # causes breakdown
>>> print('text = %r' % (text,))
>>> sentence_break = not ut.get_argflag('--nobreak')
>>> wrapped_text = format_single_paragraph_sentences(text, debug=True, sentence_break=sentence_break)
>>> result = ('wrapped_text =\n%s' % (str(wrapped_text),))
>>> print(result)
utool.util_str.format_text_as_docstr(text)[source]

CommandLine:

python  ~/local/vim/rc/pyvim_funcs.py  --test-format_text_as_docstr

Example

>>> # DISABLE_DOCTEST
>>> from pyvim_funcs import *  # NOQA
>>> text = testdata_text()
>>> formated_text = format_text_as_docstr(text)
>>> result = ('formated_text = \n%s' % (str(formated_text),))
>>> print(result)
utool.util_str.func_callsig(func, with_name=True)[source]

String of function call signature

Parameters:func (function) – live python function
Returns:callsig
Return type:str

CommandLine:

python -m utool.util_str --exec-func_callsig

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> func = func_str
>>> callsig = func_callsig(func)
>>> result = str(callsig)
>>> print(result)
func_str(func, args, kwargs, type_aliases, packed, packkw)
utool.util_str.func_defsig(func, with_name=True)[source]

String of function definition signature

Parameters:func (function) – live python function
Returns:defsig
Return type:str

CommandLine:

python -m utool.util_str --exec-func_defsig

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> func = func_str
>>> defsig = func_defsig(func)
>>> result = str(defsig)
>>> print(result)
func_str(func, args=[], kwargs={}, type_aliases=[], packed=False, packkw=None)
utool.util_str.func_str(func, args=[], kwargs={}, type_aliases=[], packed=False, packkw=None)[source]

string representation of function definition

Returns:

a representation of func with args, kwargs, and type_aliases
Return type:

str
Parameters:
  • func (function) –
  • args (list) – argument values (default = [])
  • kwargs (dict) – kwargs values (default = {})
  • type_aliases (list) – (default = [])
  • packed (bool) – (default = False)
  • packkw (None) – (default = None)
Returns:

func_str
Return type:

str

CommandLine:

python -m utool.util_str --exec-func_str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> func = byte_str
>>> args = [1024, 'MB']
>>> kwargs = dict(precision=2)
>>> type_aliases = []
>>> packed = False
>>> packkw = None
>>> _str = func_str(func, args, kwargs, type_aliases, packed, packkw)
>>> result = _str
>>> print(result)
byte_str(1024, 'MB', precision=2)
utool.util_str.get_bytes(nUnits, unit)[source]
utool.util_str.get_callable_name(func)[source]

Works on must functionlike objects including str, which has no func_name

Parameters:func –
Returns:
Return type:?

CommandLine:

python -m utool.util_str --exec-get_callable_name

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> func = len
>>> result = get_callable_name(func)
>>> print(result)
len
utool.util_str.get_colored_diff(text)[source]
utool.util_str.get_freespace_str(dir_=u'.')[source]

returns string denoting free disk space in a directory

utool.util_str.get_indentation(line_)[source]

returns the number of preceding spaces

utool.util_str.get_itemstr_list(list_, strvals=False, newlines=True, recursive=True, indent_=u'', precision=None, label_list=None, with_comma=True, **listkws)[source]

TODO: have this replace dict_itemstr list or at least most functionality in it. have it make two itemstr lists over keys and values and then combine them.

utool.util_str.get_minimum_indentation(text)[source]

returns the number of preceding spaces

Args:
text (str): unicode text
Returns:
int: indentation
CommandLine:
python -m utool.util_str –exec-get_minimum_indentation –show
Example:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> text = '    foo
bar’
>>> result = get_minimum_indentation(text)
>>> print(result)
3
utool.util_str.get_textdiff(text1, text2, num_context_lines=0, ignore_whitespace=False)[source]

Uses difflib to return a difference string between two similar texts

References

http://www.java2s.com/Code/Python/Utility/IntelligentdiffbetweentextfilesTimPeters.htm

Parameters:
  • text1 –
  • text2 –
Returns:
Return type:

?

CommandLine:

python -m utool.util_str --test-get_textdiff:1
python -m utool.util_str --test-get_textdiff:0

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> # build test data
>>> text1 = 'one\ntwo\nthree'
>>> text2 = 'one\ntwo\nfive'
>>> # execute function
>>> result = get_textdiff(text1, text2)
>>> # verify results
>>> print(result)
- three
+ five
Example2:
>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> # build test data
>>> text1 = 'one\ntwo\nthree\n3.1\n3.14\n3.1415\npi\n3.4\n3.5\n4'
>>> text2 = 'one\ntwo\nfive\n3.1\n3.14\n3.1415\npi\n3.4\n4'
>>> # execute function
>>> num_context_lines = 1
>>> result = get_textdiff(text1, text2, num_context_lines)
>>> # verify results
>>> print(result)
utool.util_str.get_unix_timedelta_str(unixtime_diff)[source]

string representation of time deltas

utool.util_str.highlight_code(text, lexer_name=u'python')[source]
utool.util_str.highlight_regex(str_, pat, reflags=0, color=u'red')[source]

FIXME Use pygments instead

utool.util_str.highlight_text(text, lexer_name=u'python', **kwargs)[source]
SeeAlso:
color_text
utool.util_str.horiz_string(*args, **kwargs)[source]

Horizontally concatenates strings reprs preserving indentation

Concats a list of objects ensuring that the next item in the list is all the way to the right of any previous items.

CommandLine:

python -m utool.util_str --test-horiz_string
Example1:
>>> # ENABLE_DOCTEST
>>> # Pretty printing of matrices demo / test
>>> import utool
>>> import numpy as np
>>> # Wouldn't it be nice if we could print this operation easily?
>>> B = np.array(((1, 2), (3, 4)))
>>> C = np.array(((5, 6), (7, 8)))
>>> A = B.dot(C)
>>> # Eg 1:
>>> result = (utool.hz_str('A = ', A, ' = ', B, ' * ', C))
>>> print(result)
A = [[19 22]  = [[1 2]  * [[5 6]
     [43 50]]    [3 4]]    [7 8]]
Exam2:
>>> # Eg 2:
>>> str_list = ['A = ', str(B), ' * ', str(C)]
>>> horizstr = (utool.horiz_string(*str_list))
>>> result = (horizstr)
>>> print(result)
A = [[1 2]  * [[5 6]
     [3 4]]    [7 8]]
utool.util_str.hz_str(*args, **kwargs)

Horizontally concatenates strings reprs preserving indentation

Concats a list of objects ensuring that the next item in the list is all the way to the right of any previous items.

CommandLine:

python -m utool.util_str --test-horiz_string
Example1:
>>> # ENABLE_DOCTEST
>>> # Pretty printing of matrices demo / test
>>> import utool
>>> import numpy as np
>>> # Wouldn't it be nice if we could print this operation easily?
>>> B = np.array(((1, 2), (3, 4)))
>>> C = np.array(((5, 6), (7, 8)))
>>> A = B.dot(C)
>>> # Eg 1:
>>> result = (utool.hz_str('A = ', A, ' = ', B, ' * ', C))
>>> print(result)
A = [[19 22]  = [[1 2]  * [[5 6]
     [43 50]]    [3 4]]    [7 8]]
Exam2:
>>> # Eg 2:
>>> str_list = ['A = ', str(B), ' * ', str(C)]
>>> horizstr = (utool.horiz_string(*str_list))
>>> result = (horizstr)
>>> print(result)
A = [[1 2]  * [[5 6]
     [3 4]]    [7 8]]
utool.util_str.indent(str_, indent=u' ')[source]

Indents a block of text

Args:
str_ (str): indent (str): (default = ‘ ‘) TODO rename to indent_ or rename func
Returns:
str:
CommandLine:
python -m utool.util_str –test-indent
Example:
>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> str_ = 'foobar
bazbiz’
>>> indent = '    '
>>> result = indent(str_, indent)
>>> print(result)
utool.util_str.indent_list(indent, list_)[source]
utool.util_str.indent_rest(str_, indent=u' ')[source]

TODO fix name Indents every part of the string except the beginning SeeAlso: ibeis/templates/generate_notebook.py

utool.util_str.indentcat(str1, str2, indent=u' ')[source]
utool.util_str.indentjoin(strlist, indent=u'\n ', suffix=u'')[source]

Convineince indentjoin

similar to ‘n ‘.join(strlist) but indent is also prefixed

Parameters:
  • strlist –
  • indent (str) –
  • suffix (str) –
Returns:

joined list
Return type:

str
utool.util_str.insert_before_sentinal(text, repl_, sentinal)[source]
utool.util_str.insert_block_between_lines(text, row1, row2, line_list, inplace=False)[source]
utool.util_str.is_byte_encoded_unicode(str_)[source]
utool.util_str.is_url(str_)[source]

heuristic check if str is url formatted

utool.util_str.joins(string, list_, with_head=True, with_tail=False, tostrip=u'\n')[source]
utool.util_str.list_str(list_, indent_=u'', newlines=1, nobraces=False, nl=None, truncate=False, truncatekw={}, label_list=None, packed=False, nobr=None, **listkw)[source]
Parameters:
  • list (list) –
  • indent (str) – (default = ‘’)
  • newlines (int) – (default = 1)
  • nobraces (bool) – (default = False)
  • nl (None) – alias for newlines (default = None)
  • truncate (bool) – (default = False)
  • truncatekw (dict) – (default = {})
  • label_list (list) – (default = None)
  • packed (bool) – if true packs braces close to body (default = False)
Kwargs:
strvals, recursive, precision, with_comma
Returns:retstr
Return type:str

CommandLine:

python -m utool.util_str --test-list_str
python -m utool.util_str --exec-list_str --truncate=True
python -m utool.util_str --exec-list_str --truncate=0

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> list_ = [[(('--verbose-qt', '--verbqt'), 1, False, ''),
...     (('--verbose-qt', '--verbqt'), 1, False, ''), (('--verbose-qt',
...     '--verbqt'), 1, False, ''), (('--verbose-qt', '--verbqt'), 1,
...     False, '')], [(['--nodyn'], 1, False, ''), (['--nodyn'], 1, False,
...     '')]]
>>> indent_ = ''
>>> newlines = 2
>>> truncate = ut.get_argval('--truncate', type_=None, default=False)
>>> nobraces = False
>>> nl = None
>>> result = list_str(list_, indent_, newlines, nobraces, nl,
>>>                   truncate=truncate, truncatekw={'maxlen': 10})
>>> print(result)
[
    [
        (('--verbose-qt', '--verbqt'), 1, False, ''),
        (('--verbose-qt', '--verbqt'), 1, False, ''),
        (('--verbose-qt', '--verbqt'), 1, False, ''),
        (('--verbose-qt', '--verbqt'), 1, False, ''),
    ],
    [
        (['--nodyn'], 1, False, ''),
        (['--nodyn'], 1, False, ''),
    ],
]
utool.util_str.list_str_summarized(list_, list_name, maxlen=5)[source]

prints the list members when the list is small and the length when it is large

utool.util_str.listinfo_str(list_)[source]
utool.util_str.long_fname_format(fmt_str, fmt_dict, hashable_keys=[], max_len=64, hashlen=16, ABS_MAX_LEN=255, hack27=False)[source]

Formats a string and hashes certain parts if the resulting string becomes too long. Used for making filenames fit onto disk.

Parameters:
  • fmt_str (str) – format of fname
  • fmt_dict (str) – dict to format fname with
  • hashable_keys (list) – list of dict keys you are willing to have hashed
  • max_len (int) – tries to fit fname into this length
  • ABS_MAX_LEN (int) – throws AssertionError if fname over this length

CommandLine:

python -m utool.util_str --exec-long_fname_format

Example

>>> # ENABLE_DOCTET
>>> import utool as ut
>>> fmt_str = 'qaid={qaid}_res_{cfgstr}_quuid={quuid}'
>>> quuid_str = 'blahblahblahblahblahblah'
>>> cfgstr = 'big_long_string__________________________________'
>>> qaid = 5
>>> fmt_dict = dict(cfgstr=cfgstr, qaid=qaid, quuid=quuid_str)
>>> hashable_keys = ['cfgstr', 'quuid']
>>> max_len = 64
>>> hashlen = 8
>>> fname0 = ut.long_fname_format(fmt_str, fmt_dict, max_len=None)
>>> fname1 = ut.long_fname_format(fmt_str, fmt_dict, hashable_keys,
>>>                                  max_len=64, hashlen=8)
>>> fname2 = ut.long_fname_format(fmt_str, fmt_dict, hashable_keys, max_len=42,
>>>                         hashlen=8)
>>> result = fname0 + '\n' + fname1 + '\n' + fname2
>>> print(result)
qaid=5_res_big_long_string___________________________________quuid=blahblahblahblahblahblah
qaid=5_res_kjrok785_quuid=blahblahblahblahblahblah
qaid=5_res_du1&i&5l_quuid=euuaxoyi
utool.util_str.lorium_ipsum()[source]
utool.util_str.msgblock(key, text)[source]

puts text inside a visual ascii block

utool.util_str.multi_replace(str_, search_list, repl_list)[source]

Performs multiple replace functions foreach item in search_list and repl_list.

Parameters:
  • str (str) – string to search
  • search_list (list) – list of search strings
  • repl_list (list or str) – one or multiple replace strings
Returns:

str_
Return type:

str

CommandLine:

python -m utool.util_str --exec-multi_replace

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> str_ = 'foo. bar: baz; spam-eggs --- eggs+spam'
>>> search_list = ['.', ':', '---']
>>> repl_list = '@'
>>> str_ = multi_replace(str_, search_list, repl_list)
>>> result = ('str_ = %s' % (str(str_),))
>>> print(result)
str_ = foo@ bar@ baz; spam-eggs @ eggs+spam
utool.util_str.newlined_list(list_, joinstr=u', ', textwidth=160)[source]

Converts a list to a string but inserts a new line after textwidth chars DEPRICATE

utool.util_str.number_text_lines(text)[source]
Parameters:text (str) –
Returns:text_with_lineno - string with numbered lines
Return type:str
utool.util_str.numeric_str(num, precision=None, **kwargs)[source]
Parameters:
  • num (scalar or array) –
  • precision (int) –
Returns:
Return type:

str

CommandLine:

python -m utool.util_str --test-numeric_str

References

http://stackoverflow.com/questions/4541155/check-if-a-number-is-int-or-float

Notes

isinstance(np.array([3], dtype=np.uint8)[0], numbers.Integral) isinstance(np.array([3], dtype=np.int32)[0], numbers.Integral) isinstance(np.array([3], dtype=np.uint64)[0], numbers.Integral) isinstance(np.array([3], dtype=object)[0], numbers.Integral) isinstance(np.array([3], dtype=np.float32)[0], numbers.Integral) isinstance(np.array([3], dtype=np.float64)[0], numbers.Integral)

CommandLine:

python -m utool.util_str --test-numeric_str

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> precision = 2
>>> result = [numeric_str(num, precision) for num in [1, 2.0, 3.43343,4432]]
>>> print(result)
['1', '2.00', '3.43', '4432']
utool.util_str.numpy_str(arr, strvals=False, precision=None, pr=None, force_dtype=True, with_dtype=None, suppress_small=None, max_line_width=None, threshold=None, **kwargs)[source]

suppress_small = False turns off scientific representation

utool.util_str.numpy_str2(arr, **kwargs)[source]
utool.util_str.obj_str(obj_, **kwargs)[source]

DEPRICATE in favor of repr2

utool.util_str.order_of_magnitude_str(num, base=10.0, prefix_list=None, exponent_list=None, suffix=u'', prefix=None)[source]

TODO: Rewrite byte_str to use this func :returns: str

utool.util_str.pack_into(text, textwidth=160, breakchars=u' ', break_words=True, newline_prefix=u'', wordsep=u' ', remove_newlines=True)[source]

DEPRICATE IN FAVOR OF textwrap.wrap

TODO: Look into textwrap.wrap

Inserts newlines into a string enforcing a maximum textwidth. Similar to vim’s gq command in visual select mode.

breakchars is a string containing valid characters to insert a newline before or after.

break_words is True if words are allowed to be split over multiple lines.

all inserted newlines are prefixed with newline_prefix

#FIXME:

Example

>>> text = "set_image_uris(ibs<139684018194000>, [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], [u'66ec193a-1619-b3b6-216d-1784b4833b61.jpg', u'd8903434-942f-e0f5-d6c2-0dcbe3137bf7.jpg', u'b73b72f4-4acb-c445-e72c-05ce02719d3d.jpg', u'0cd05978-3d83-b2ee-2ac9-798dd571c3b3.jpg', u'0a9bc03d-a75e-8d14-0153-e2949502aba7.jpg', u'2deeff06-5546-c752-15dc-2bd0fdb1198a.jpg', u'a9b70278-a936-c1dd-8a3b-bc1e9a998bf0.png', u'42fdad98-369a-2cbc-67b1-983d6d6a3a60.jpg', u'c459d381-fd74-1d99-6215-e42e3f432ea9.jpg', u'33fd9813-3a2b-774b-3fcc-4360d1ae151b.jpg', u'97e8ea74-873f-2092-b372-f928a7be30fa.jpg', u'588bc218-83a5-d400-21aa-d499832632b0.jpg', u'163a890c-36f2-981e-3529-c552b6d668a3.jpg'], ) "  # NOQA
>>> textwidth = 160
>>> breakchars = ' '
>>> break_words = True
>>> newline_prefix = '    '
>>> wordsep = ' '
>>> packstr1 = pack_into(text, textwidth, breakchars, break_words, newline_prefix, wordsep)
>>> break_words = False
>>> packstr2 = pack_into(text, textwidth, breakchars, break_words, newline_prefix, wordsep)
>>> print(packstr1)
>>> print(packstr2)

CommandLine:

python -c "import utool" --dump-utool-init
utool.util_str.packstr(instr, textwidth=160, breakchars=u' ', break_words=True, newline_prefix=u'', indentation=u'', nlprefix=None, wordsep=u' ', remove_newlines=True)[source]

alias for pack_into. has more up to date kwargs

utool.util_str.packtext(text, width=80)[source]
Parameters:text (str) –

CommandLine:

python -m utool.util_str --exec-pack_paragraph --show

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> width = 80
>>> text = lorium_ipsum()
>>> result = packtext(text)
>>> print(result)
utool.util_str.padded_str_range(start, end)[source]

Builds a list of (end - start) strings padded with zeros

utool.util_str.parse_bytes(bytes_str)[source]

uint8_size = ut.parse_bytes(‘1B’) image_size = ut.parse_bytes(‘3.5MB’) float32_size = ut.parse_bytes(‘32bit’) desc_size = 128 * uint8_size kpts_size = 6 * float32_size chip_size = ut.parse_bytes(‘400 KB’) probchip_size = ut.parse_bytes(‘50 KB’) nImgs = 80000 # 80,000 nAnnots = nImgs * 2 desc_per_img = 3000 size_stats = {

‘image’: nImgs * image_size, ‘chips’: nAnnots * chip_size, ‘probchips’: nAnnots * probchip_size, ‘desc’: nAnnots * desc_size * desc_per_img, ‘kpts’: nAnnots * kpts_size * desc_per_img,

} print(ut.repr3(ut.map_dict_vals(ut.byte_str2, size_stats), align=True)) print(‘total = ‘ + ut.byte_str2(sum(size_stats.values())))

utool.util_str.percent_str(pcnt)[source]

Depricate

utool.util_str.pluralize(wordtext, num, plural_suffix=u's')[source]
utool.util_str.quantity_str(typestr, num, plural_suffix=u's')[source]
utool.util_str.regex_reconstruct_split(pattern, text, debug=False)[source]
utool.util_str.remove_chars(str_, char_list)[source]

removes all chars in char_list from str_

Parameters:
  • str (str) –
  • char_list (list) –
Returns:

outstr
Return type:

str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> str_ = '1, 2, 3, 4'
>>> char_list = [',']
>>> result = remove_chars(str_, char_list)
>>> print(result)
1 2 3 4
utool.util_str.remove_doublspaces(text)[source]
utool.util_str.remove_vowels(str_)[source]

strips all vowels from a string

utool.util_str.replace_between_tags(text, repl_, start_tag, end_tag=None)[source]

Replaces text between sentinal lines in a block of text.

Parameters:
  • text (str) –
  • repl (str) –
  • start_tag (str) –
  • end_tag (str) – (default=None)
Returns:

new_text
Return type:

str

CommandLine:

python -m utool.util_str --exec-replace_between_tags

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> text = ut.codeblock(
    '''
    class:
        # <FOO>
        bar
        # </FOO>
        baz
    ''')
>>> repl_ = 'spam'
>>> start_tag = '# <FOO>'
>>> end_tag = '# </FOO>'
>>> new_text = replace_between_tags(text, repl_, start_tag, end_tag)
>>> result = ('new_text =\n%s' % (str(new_text),))
>>> print(result)
new_text =
class:
    # <FOO>
spam
    # </FOO>
    baz
utool.util_str.replace_nonquoted_text(text, search_list, repl_list)[source]

WARNING: this function is not safely implemented. It can break of searching for single characters or underscores. Depends on utool.modify_quoted_strs which is also unsafely implemented

Parameters:
  • text –
  • search_list (list) –
  • repl_list (list) –

Example

>>> from utool.util_str import *  # NOQA
>>> text = '?'
>>> search_list = '?'
>>> repl_list = '?'
>>> result = replace_nonquoted_text(text, search_list, repl_list)
>>> print(result)
utool.util_str.repr2(obj_, **kwargs)[source]

Use in favor of obj_str. Attempt to replace repr more configurable pretty version that works the same in both 2 and 3

utool.util_str.repr3(obj_, **kwargs)[source]
utool.util_str.reprfunc(val, precision=None)[source]
utool.util_str.scalar_str(val, precision=None, max_precision=None)[source]
utool.util_str.seconds_str(num, prefix=None)[source]
Returns:str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> import utool as ut
>>> num_list = sorted([4.2 / (10.0 ** exp_)
>>>                    for exp_ in range(-13, 13, 4)])
>>> secstr_list = [seconds_str(num, prefix=None) for num in num_list]
>>> result = (', '.join(secstr_list))
>>> print(result)
0.04 ns, 0.42 us, 4.20 ms, 0.04 ks, 0.42 Ms, 4.20 Gs, 42.00 Ts
utool.util_str.singular_string(str_, plural_suffix=u's', singular_suffix=u'')[source]

tries to use english grammar to make a string singular very naive implementation. will break often

utool.util_str.str2(obj)[source]
utool.util_str.str_between(str_, startstr, endstr)[source]

gets substring between two sentianl strings

utool.util_str.strip_ansi(text)[source]
utool.util_str.testdata_text(num=1)[source]
utool.util_str.textblock(multiline_text)[source]
Parameters:block_str (str) –

CommandLine:

python -m utool.util_str --test-textblock

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> # build test data
>>> multiline_text = ''' a big string
    that should be layed out flat
    yet still provide nice python
    code that doesnt go too far over
    80 characters.
Two newlines should be respected though ‘’‘
>>> # execute function
>>> new_text = textblock(multiline_text)
>>> # verify results
>>> result = new_text
>>> print(result)
utool.util_str.theta_str(theta, taustr=u'*2pi', fmtstr=u'{coeff:, .1f}{taustr}')[source]

Format theta so it is interpretable in base 10

Parameters:
  • theta (float) –
  • taustr (str) – default 2pi
Returns:

theta_str - the angle in tau units
Return type:

str
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> theta = 3.1415
>>> result = theta_str(theta)
>>> print(result)
0.5*2pi
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> theta = 6.9932
>>> taustr = 'tau'
>>> result = theta_str(theta, taustr)
>>> print(result)
1.1tau
utool.util_str.to_camel_case(underscore_case, mixed=False)[source]
Parameters:underscore_case –
Returns:camel_case_str
Return type:str

CommandLine:

python -m utool.util_str --exec-to_camel_case

References

https://en.wikipedia.org/wiki/CamelCase

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> underscore_case = 'underscore_funcname'
>>> camel_case_str = to_camel_case(underscore_case)
>>> result = ('camel_case_str = %s' % (str(camel_case_str),))
>>> print(result)
camel_case_str = UnderscoreFuncname
utool.util_str.to_title_caps(underscore_case)[source]
Parameters:underscore_case –
Returns:title_str
Return type:str

CommandLine:

python -m utool.util_str --exec-to_title_caps

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> underscore_case = 'the_foo_bar_func'
>>> title_str = to_title_caps(underscore_case)
>>> result = ('title_str = %s' % (str(title_str),))
>>> print(result)
title_str = The Foo Bar Func
utool.util_str.to_underscore_case(camelcase_str)[source]

References

http://stackoverflow.com/questions/1175208/convert-camelcase

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_str import *  # NOQA
>>> camelcase_str = 'UnderscoreFuncname'
>>> camel_case_str = to_underscore_case(camelcase_str)
>>> result = ('underscore_str = %s' % (str(camel_case_str),))
>>> print(result)
underscore_str = underscore_funcname
utool.util_str.trunc_repr(obj)[source]
utool.util_str.truncate_str(str_, maxlen=110, truncmsg=u' ~~~TRUNCATED~~~ ')[source]

Removes the middle part of any string over maxlen characters.

utool.util_str.tupstr(tuple_)[source]

maps each item in tuple to a string and doesnt include parens

utool.util_str.unformat_text_as_docstr(formated_text)[source]

CommandLine:

python  ~/local/vim/rc/pyvim_funcs.py  --test-unformat_text_as_docstr

Example

>>> # DISABLE_DOCTEST
>>> from pyvim_funcs import *  # NOQA
>>> text = testdata_text()
>>> formated_text = format_text_as_docstr(text)
>>> unformated_text = unformat_text_as_docstr(formated_text)
>>> result = ('unformated_text = \n%s' % (str(unformated_text),))
>>> print(result)
utool.util_str.unindent(string)[source]

Unindent a block of text

Alias for textwrap.dedent

utool.util_str.varinfo_str(varval, varname, onlyrepr=False, canshowrepr=True, varcolor=u'yellow', colored=True)[source]
utool.util_str.verts_str(verts, pad=1)[source]

makes a string from a list of integer verticies

utool.util_sysreq module

utool.util_sysreq.ensure_in_pythonpath(dname)[source]
utool.util_sysreq.get_site_packages_dir()[source]

Notes

It seems IPython does not respect virtual environments properly. TODO: find a solution http://stackoverflow.com/questions/7335992/ipython-and-virtualenv-ignoring-site-packages

utool.util_sysreq.in_virtual_env()[source]

returns True if you are running inside a python virtual environment. (DOES NOT WORK IF IN IPYTHON AND USING A VIRTUALENV)

sys.prefix gives the location of the virtualenv

Notes

It seems IPython does not respect virtual environments properly. TODO: find a solution http://stackoverflow.com/questions/7335992/ipython-and-virtualenv-ignoring-site-packages

References

http://stackoverflow.com/questions/1871549/python-determine-if-running-inside-virtualenv

utool.util_sysreq.is_running_as_root()[source]

References

http://stackoverflow.com/questions/5721529/running-python-script-as-root-with-sudo-what-is-the-username-of-the-effectiv http://stackoverflow.com/questions/2806897/what-is-the-best-practices-for-checking-if-the-user-of-a-python-script-has-root

utool.util_sysreq.locate_path(dname, recurse_down=True)[source]

Search for a path

utool.util_tags module

utool.util_tags.filterflags_general_tags(tags_list, has_any=None, has_all=None, has_none=None, min_num=None, max_num=None, any_startswith=None, any_endswith=None, any_match=None, none_match=None, logic=u'and', ignore_case=True)[source]

maybe integrate into utool? Seems pretty general

Parameters:
  • tags_list (list) –
  • has_any (None) – (default = None)
  • has_all (None) – (default = None)
  • min_num (None) – (default = None)
  • max_num (None) – (default = None)

CommandLine:

python -m utool.util_tags --exec-filterflags_general_tags
python -m utool.util_tags --exec-filterflags_general_tags:0  --helpx
python -m utool.util_tags --exec-filterflags_general_tags:0
python -m utool.util_tags --exec-filterflags_general_tags:0  --none_match n
python -m utool.util_tags --exec-filterflags_general_tags:0  --has_none=n,o
python -m utool.util_tags --exec-filterflags_general_tags:1
python -m utool.util_tags --exec-filterflags_general_tags:2
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_tags import *  # NOQA
>>> import utool as ut
>>> tags_list = [['v'], [], ['P'], ['P', 'o'], ['n', 'o',], [], ['n', 'N'], ['e', 'i', 'p', 'b', 'n'], ['q', 'v'], ['n'], ['n'], ['N']]
>>> kwargs = ut.argparse_dict(ut.get_kwdefaults2(filterflags_general_tags), type_hint=list)
>>> print('kwargs = %r' % (kwargs,))
>>> flags = filterflags_general_tags(tags_list, **kwargs)
>>> print(flags)
>>> result = ut.compress(tags_list, flags)
>>> print('result = %r' % (result,))
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_tags import *  # NOQA
>>> import utool as ut
>>> tags_list = [['v'], [], ['P'], ['P'], ['n', 'o',], [], ['n', 'N'], ['e', 'i', 'p', 'b', 'n'], ['n'], ['n'], ['N']]
>>> has_all = 'n'
>>> min_num = 1
>>> flags = filterflags_general_tags(tags_list, has_all=has_all, min_num=min_num)
>>> result = ut.compress(tags_list, flags)
>>> print('result = %r' % (result,))
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_tags import *  # NOQA
>>> import utool as ut
>>> tags_list = [['vn'], ['vn', 'no'], ['P'], ['P'], ['n', 'o',], [], ['n', 'N'], ['e', 'i', 'p', 'b', 'n'], ['n'], ['n', 'nP'], ['NP']]
>>> kwargs = {
>>>     'any_endswith': 'n',
>>>     'any_match': None,
>>>     'any_startswith': 'n',
>>>     'has_all': None,
>>>     'has_any': None,
>>>     'has_none': None,
>>>     'max_num': 3,
>>>     'min_num': 1,
>>>     'none_match': ['P'],
>>> }
>>> flags = filterflags_general_tags(tags_list, **kwargs)
>>> filtered = ut.compress(tags_list, flags)
>>> result = ('result = %s' % (ut.repr2(filtered),))
result = [['vn', 'no'], ['n', 'o'], ['n', 'N'], ['n'], ['n', 'nP']]

utool.util_tests module

Helpers for tests

This module contains a more sane reimplementation of doctest functionality. (I.E. asserts work and you don’t have to worry about stdout mucking things up) The code isn’t super clean though due to time constriaints. Many functions probably belong elsewhere and the parsers need a big cleanup.

TODO:
  • report the line of the doctest in the file when reporting errors as well as
the relative line
  • restructure so there is a test collection step, a filtering step, and an execution step
  • Fix finding tests when running with @profile
exception utool.util_tests.ExitTestException[source]

Bases: exceptions.Exception

class utool.util_tests.ModuleDoctestTup(enabled_testtup_list, frame_fpath, all_testflags, module)

Bases: tuple

__getnewargs__()

Return self as a plain tuple. Used by copy and pickle.

__getstate__()

Exclude the OrderedDict from pickling

__repr__()

Return a nicely formatted representation string

all_testflags

Alias for field number 2

enabled_testtup_list

Alias for field number 0

frame_fpath

Alias for field number 1

module

Alias for field number 3

class utool.util_tests.TestTuple(name, num, src, want, flag, tags=None, frame_fpath=None, exec_mode=False)[source]

Bases: utool.util_dev.NiceRepr

Simple container for test objects to replace old tuple format exec mode specifies if the test is being run as a script

utool.util_tests.doctest_funcs(testable_list=None, check_flags=True, module=None, allexamples=None, needs_enable=None, strict=False, verbose=True, return_error_report=False, seen_=None)[source]

Main entry point into utools main module doctest harness Imports a module and checks flags for the function to run Depth 1)

Parameters:
  • testable_list (list) –
  • check_flags (bool) – Force checking of the –test- and –exec- flags
  • module (None) –
  • allexamples (None) –
  • needs_enable (None) –
Returns:

(nPass, nTotal, failed_cmd_list)
Return type:

tuple

CommandLine:

python -m ibeis.algo.preproc.preproc_chip --all-examples

References

http://legacy.python.org/dev/peps/pep-0338/ https://docs.python.org/2/library/runpy.html

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_tests import *  # NOQA
>>> testable_list = []
>>> check_flags = True
>>> module = None
>>> allexamples = None
>>> needs_enable = None
>>> # careful might infinitely recurse
>>> (nPass, nTotal) = doctest_funcs(testable_list, check_flags, module,
...                                 allexamples, needs_enable)
>>> print((nPass, nTotal))
utool.util_tests.doctest_module_list(module_list)[source]

Runs many module tests

Entry point for batch run Depth 0)

utool.util_tests.doctest_was_requested()[source]

lets a __main__ codeblock know that util_test should do its thing

utool.util_tests.execute_doctest(func, testnum=0, module=None)[source]

Execute a function doctest. Can optionaly specify func name and module to run from ipython notebooks.

Example

>>> from utool.util_tests import *  # NOQA
IPython:
import utool as ut ut.execute_doctest(func=’dummy_example_depcacahe’, module=’dtool.example_depcache’)
utool.util_tests.find_doctestable_modnames(dpath_list=None, exclude_doctests_fnames=[], exclude_dirs=[])[source]

Tries to find files with a call to ut.doctest_funcs in the __main__ part

utool.util_tests.find_testfunc(module, test_funcname, ignore_prefix=[], ignore_suffix=[], func_to_module_dict={})[source]
utool.util_tests.find_untested_modpaths(dpath_list=None, exclude_doctests_fnames=[], exclude_dirs=[])[source]
utool.util_tests.get_doctest_examples(func_or_class)[source]

Depth 3) called by get_module_doctest_tup

Parameters:func_or_class (function) –
Returns:tuple – example_list, want_list
Return type:list, list

CommandLine:

python -m utool.util_tests --test-get_doctest_examples
Example0:
>>> # ENABLE_DOCTEST
>>> from utool.util_tests import *  # NOQA
>>> func_or_class = get_doctest_examples
>>> tup  = get_doctest_examples(func_or_class)
>>> testsrc_list, testwant_list, testlinenum_list, func_lineno, docstr = tup
>>> result = str(len(testsrc_list) + len(testwant_list))
>>> print(testsrc_list)
>>> print(testlinenum_list)
>>> print(func_lineno)
>>> print(testwant_list)
>>> print(result)
6
Example1:
>>> # ENABLE_DOCTEST
>>> from utool.util_tests import *  # NOQA
>>> import utool as ut
>>> func_or_class = ut.tryimport
>>> tup = get_doctest_examples(func_or_class)
>>> testsrc_list, testwant_list, testlinenum_list, func_lineno, docstr = tup
>>> result = str(len(testsrc_list) + len(testwant_list))
>>> print(testsrc_list)
>>> print(testlinenum_list)
>>> print(func_lineno)
>>> print(testwant_list)
>>> print(result)
4
Example2:
>>> # DISABLE_DOCTEST
>>> from utool.util_tests import *  # NOQA
>>> import ibeis
>>> func_or_class = ibeis.control.manual_annot_funcs.add_annots
>>> tup = get_doctest_examples(func_or_class)
>>> testsrc_list, testwant_list, testlinenum_list, func_lineno, docstr = tup
>>> result = str(len(testsrc_list) + len(testwant_list))
>>> print(testsrc_list)
>>> print(testlinenum_list)
>>> print(func_lineno)
>>> print(testwant_list)
>>> print(result)
2
utool.util_tests.get_module_doctest_tup(testable_list=None, check_flags=True, module=None, allexamples=None, needs_enable=None, N=0, verbose=True, testslow=False)[source]

Parses module for testable doctesttups Depth 2)

Parameters:
  • testable_list (list) – a list of functions (default = None)
  • check_flags (bool) – (default = True)
  • module (None) – (default = None)
  • allexamples (None) – (default = None)
  • needs_enable (None) – (default = None)
  • N (int) – (default = 0)
  • verbose (bool) – verbosity flag(default = True)
  • testslow (bool) – (default = False)
Returns:

(enabled_testtup_list, frame_fpath, all_testflags, module) enabled_testtup_list (list): a list of testtup

testtup (tuple): (name, num, src, want, flag) describes a valid doctest in the module

name (str): test name num (str): test number of the module / function / class / method src (str): test source code want (str): expected test result flag (str): a valid commandline flag to enable this test

frame_fpath (str):

module fpath that will be tested

module (module):

the actual module that will be tested

all_testflags (list):

the command line arguments that will enable different tests

exclude_inherited (bool): does not included tests defined in other modules
Return type:

ModuleDoctestTup

CommandLine:

python -m utool.util_tests --exec-get_module_doctest_tup

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_tests import *  # NOQA
>>> import utool as ut
>>> #testable_list = [ut.util_import.package_contents]
>>> testable_list = None
>>> check_flags = False
>>> module = ut.util_cplat
>>> allexamples = False
>>> needs_enable = None
>>> N = 0
>>> verbose = True
>>> testslow = False
>>> mod_doctest_tup = get_module_doctest_tup(testable_list, check_flags, module, allexamples, needs_enable, N, verbose, testslow)
>>> result = ('mod_doctest_tup = %s' % (ut.list_str(mod_doctest_tup, nl=4),))
>>> print(result)
utool.util_tests.get_module_testlines(module_list, remove_pyc=True, verbose=True, pythoncmd=None)[source]

Builds test commands for autogen tests called by autogen test scripts

utool.util_tests.get_package_testables(module=None, **tagkw)[source]

New command that should eventually be used intead of old stuff?

Parameters:
  • module_list (list) – (default = None)
  • test_flags (None) – (default = None)

CommandLine:

python -m utool.util_tests --exec-get_package_testables --show --mod ibeis
python -m utool.util_tests --exec-get_package_testables --show --mod utool --tags SCRIPT
python -m utool.util_tests --exec-get_package_testables --show --mod utool --tags ENABLE

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_tests import *  # NOQA
>>> import utool as ut
>>> has_any = ut.get_argval('--tags', type_=str, default=None)
>>> module = ut.get_argval('--mod', default='utool')
>>> test_tuples = get_package_testables(module, has_any=has_any)
>>> result = ut.repr3(test_tuples)
>>> print(result)
>>> #print(ut.repr3(ut.list_getattr(test_tuples, 'tags')))
utool.util_tests.main_function_tester(module, ignore_prefix=[], ignore_suffix=[], test_funcname=None, func_to_module_dict={})[source]

Allows a shorthand for __main__ packages of modules to run tests with unique function names

utool.util_tests.parse_docblocks_from_docstr(docstr)[source]

Depth 5) called by parse_doctest_from_docstr TODO: move to util_inspect

Parameters:docstr (str) –
Returns:docstr_blocks tuples [(blockname, blockstr, offset)]
Return type:list

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_tests import *  # NOQA
>>> import utool as ut
>>> #import ibeis
>>> #import ibeis.algo.hots.query_request
>>> #func_or_class = ibeis.algo.hots.query_request.QueryParams
>>> func_or_class = ut.parse_docblocks_from_docstr
>>> docstr = ut.get_docstr(func_or_class)
>>> docstr_blocks = parse_docblocks_from_docstr(docstr)
>>> result = str(docstr_blocks)
>>> print(result)
utool.util_tests.parse_doctest_from_docstr(docstr)[source]

because doctest itself doesnt do what I want it to do called by get_doctest_examples Depth 4)

CAREFUL, IF YOU GET BACK WRONG RESULTS MAKE SURE YOUR DOCSTR IS PREFFIXED WITH R

CommandLine:

python -m utool.util_tests --exec-parse_doctest_from_docstr
Setup:
>>> from utool.util_tests import *  # NOQA
>>> import utool as ut

Example

>>> # ENABLE_DOCTEST
>>> #from ibeis.algo.hots import score_normalization
>>> #func_or_class = score_normalization.cached_ibeis_score_normalizer
>>> func_or_class = parse_doctest_from_docstr
>>> docstr = ut.get_docstr(func_or_class)
>>> testsrc_list, testwant_list, testlinenum_list, func_lineno, docstr = get_doctest_examples(func_or_class)
>>> print('\n\n'.join(testsrc_list))
>>> assert len(testsrc_list) == len(testwant_list)
utool.util_tests.quit_if_noshow()[source]
utool.util_tests.read_exampleblock(docblock)[source]
utool.util_tests.run_test(func_or_testtup, return_error_report=False, *args, **kwargs)[source]

Runs the test function with success / failure printing

Parameters:func_or_testtup (func or tuple) – function or doctest tuple
Varargs/Kwargs:
Anything that needs to be passed to <func_>
utool.util_tests.show_if_requested()[source]
utool.util_tests.show_was_requested()[source]

returns True if –show is specified on the commandline or you are in IPython (and presumably want some sort of interaction

utool.util_tests.test_jedistuff()[source]

utool.util_time module

TODO: This file seems to care about timezone

TODO: Use UTC/GMT time here for EVERYTHING

References

http://www.timeanddate.com/time/aboututc.html

class utool.util_time.Timer(msg='', verbose=True, newline=True)[source]

Bases: object

Timer with-statment context object.

Example

>>> # ENABLE_DOCTEST
>>> import utool
>>> with utool.Timer('Timer test!'):
>>>     prime = utool.get_nth_prime(400)
tic()[source]
toc()[source]
utool.util_time.determine_timestamp_format(datetime_str)[source]
Parameters:datetime_str (str) –
Returns:
Return type:str

CommandLine:

python -m utool.util_time --exec-determine_timestamp_format

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> import utool as ut
>>> datetime_str_list = [
>>>     '0000:00:00 00:00:00',
>>>     '    :  :     :  :  ',
>>>     '2015:04:01 00:00:00',
>>>     '2080/04/01 00:00:00',
>>> ]
>>> result = ut.list_str([determine_timestamp_format(datetime_str)
>>>            for datetime_str in datetime_str_list])
>>> print(result)
utool.util_time.ensure_timedelta(str_or_num)[source]
utool.util_time.exiftime_to_unixtime(datetime_str, timestamp_format=None, strict=None)[source]

converts a datetime string to posixtime (unixtime)

Parameters:
  • datetime_str (str) –
  • timestamp_format (int) –
Returns:

unixtime seconds from 1970 (currently not UTC; this will change)
Return type:

int

CommandLine:

python -m utool.util_time --test-exiftime_to_unixtime

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> datetime_str = '0000:00:00 00:00:00'
>>> timestamp_format = 1
>>> result = exiftime_to_unixtime(datetime_str, timestamp_format)
>>> print(result)
-1
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> datetime_str = '2015:04:01 00:00:00'
>>> timestamp_format = 1
>>> result = exiftime_to_unixtime(datetime_str, timestamp_format)
>>> print(result)
1427846400
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> datetime_str = six.text_type('2010:10:07 19:37:03')
>>> timestamp_format = None
>>> result = exiftime_to_unixtime(datetime_str, timestamp_format)
>>> print(result)
1286480223
utool.util_time.get_datestamp(explicit=True, isutc=False)[source]
utool.util_time.get_posix_timedelta_str(posixtime)[source]

get_timedelta_str

TODO: rectify this function with get_unix_timedelta_str (unix_timedelta_str probably has better implementation)

Returns:timedelta_str, formated time string
Return type:str

CommandLine:

python -m utool.util_time --test-get_posix_timedelta_str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> posixtime_list = [-13, 10.2, 10.2 ** 2, 10.2 ** 3, 10.2 ** 4, 10.2 ** 5, 10.2 ** 8]
>>> posixtime = posixtime_list[1]
>>> timedelta_str = [get_posix_timedelta_str(posixtime) for posixtime in posixtime_list]
>>> result = (timedelta_str)
>>> print(result)
['-00:00:13', '00:00:10.20', '00:01:44.04', '00:17:41.21', '03:00:24.32', '1 days 06:40:08.08', '193 weeks 5 days 02:05:38.10']
Timeit::
import datetime # Seems like like timedelta is just faster. must be because it is builtin %timeit get_posix_timedelta_str(posixtime) %timeit str(datetime.timedelta(seconds=posixtime))
utool.util_time.get_posix_timedelta_str2(posixtime)[source]
utool.util_time.get_printable_timestamp(isutc=False)[source]
utool.util_time.get_timedelta_str(timedelta, exclude_zeros=False)[source]
Returns:timedelta_str, formated time string
Return type:str

References

http://stackoverflow.com/questions/8906926/formatting-python-timedelta-objects

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> timedelta = get_unix_timedelta(10)
>>> timedelta_str = get_timedelta_str(timedelta)
>>> result = (timedelta_str)
>>> print(result)
10 seconds
utool.util_time.get_timestamp(format_='filename', use_second=False, delta_seconds=None, isutc=False)[source]
Parameters:
  • format (str) – (tag, printable, filename, other)
  • use_second (bool) –
  • delta_seconds (None) –
Returns:

stamp
Return type:

str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> format_ = 'printable'
>>> use_second = False
>>> delta_seconds = None
>>> stamp = get_timestamp(format_, use_second, delta_seconds)
>>> print(stamp)
>>> assert len(stamp) == len('15:43:04 2015/02/24')
utool.util_time.get_timestats_dict(unixtime_list, full=True, isutc=False)[source]
utool.util_time.get_timestats_str(unixtime_list, newlines=False, full=True, isutc=False)[source]
Parameters:
  • unixtime_list (list) –
  • newlines (bool) –
Returns:

timestat_str
Return type:

str

CommandLine:

python -m utool.util_time --test-get_timestats_str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> import utool as ut
>>> unixtime_list = [0, 0 + 60*60*5 , 10+ 60*60*5, 100+ 60*60*5, 1000+ 60*60*5]
>>> newlines = True
>>> full = False
>>> timestat_str = get_timestats_str(unixtime_list, newlines, full=full, isutc=True)
>>> result = ut.align(str(timestat_str), ':')
>>> print(result)
{
    'max'  : '1970/01/01 05:16:40',
    'mean' : '1970/01/01 04:03:42',
    'min'  : '1970/01/01 00:00:00',
    'range': '5:16:40',
    'std'  : '2:02:01',
}
Example2:
>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> import utool as ut
>>> unixtime_list = [0, 0 + 60*60*5 , 10+ 60*60*5, 100+ 60*60*5, 1000+ 60*60*5, float('nan'), 0]
>>> newlines = True
>>> timestat_str = get_timestats_str(unixtime_list, newlines, isutc=True)
>>> result = ut.align(str(timestat_str), ':')
>>> print(result)
{
    'max'    : '1970/01/01 05:16:40',
    'mean'   : '1970/01/01 03:23:05',
    'min'    : '1970/01/01 00:00:00',
    'nMax'   : 1,
    'nMin'   : 2,
    'num_nan': 1,
    'range'  : '5:16:40',
    'shape'  : (7,),
    'std'    : '2:23:43',
}
utool.util_time.get_unix_timedelta(unixtime_diff)[source]
utool.util_time.get_unix_timedelta_str(unixtime_diff)[source]

TODO: rectify this function with get_posix_timedelta_str

Parameters:
  • unixtime_diff (int) – number of seconds
  • unixtime_diff –
Returns:

timestr – formated time string
Return type:

str
Returns:

timestr
Return type:

str

CommandLine:

python -m utool.util_time --test-get_unix_timedelta_str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> unixtime_diff = 0
>>> timestr = get_unix_timedelta_str(unixtime_diff)
>>> timestr_list = [get_unix_timedelta_str(_) for _ in [-9001, -1, 0, 1, 9001]]
>>> result = str(timestr_list)
>>> print(result)
['2 hours 30 minutes 1 second', '1 second', '0 seconds', '1 second', '2 hours 30 minutes 1 second']
utool.util_time.parse_timedelta_str(str_)[source]
Parameters:str (str) –
Returns:timedelta
Return type:float

CommandLine:

python -m utool.util_time --exec-parse_timedelta_str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> str_ = '24h'
>>> timedelta = parse_timedelta_str(str_)
>>> result = ('timedelta = %s' % (str(timedelta),))
>>> print(result)
timedelta = 86400.0
utool.util_time.tic(msg=None)[source]
utool.util_time.timestamp(format_='filename', use_second=False, delta_seconds=None, isutc=False)

get_timestamp

Parameters:
  • format (str) – (tag, printable, filename, other)
  • use_second (bool) –
  • delta_seconds (None) –
Returns:

stamp
Return type:

str

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_time import *  # NOQA
>>> format_ = 'printable'
>>> use_second = False
>>> delta_seconds = None
>>> stamp = get_timestamp(format_, use_second, delta_seconds)
>>> print(stamp)
>>> assert len(stamp) == len('15:43:04 2015/02/24')
utool.util_time.toc(tt, return_msg=False, write_msg=True)[source]
utool.util_time.unixtime_to_datetimeobj(unixtime, isutc=False)[source]
utool.util_time.unixtime_to_datetimestr(unixtime, timefmt='%Y/%m/%d %H:%M:%S', isutc=False)[source]

TODO: ranme to datetimestr

utool.util_time.unixtime_to_timedelta(unixtime_diff)[source]

alias for get_unix_timedelta

utool.util_type module

utool.util_type.assert_int(var, lbl=u'var')[source]
utool.util_type.bool_from_str(str_)[source]
utool.util_type.fuzzy_int(str_)[source]

lets some special strings be interpreted as ints

utool.util_type.fuzzy_subset(str_)[source]

converts a string into an argument to list_take

utool.util_type.get_homogenous_list_type(list_)[source]

Returns the best matching python type even if it is an ndarray assumes all items in the list are of the same type. does not check this

utool.util_type.get_type(var)[source]

Gets types accounting for numpy

utool.util_type.is_bool(var)[source]
utool.util_type.is_dict(var)[source]
utool.util_type.is_float(var)[source]
Parameters:var (ndarray or scalar) –
Returns:
Return type:var

CommandLine:

python -m utool.util_type --test-is_float

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_type import *  # NOQA
>>> # build test data
>>> var = np.array([1.0, 2.0, 3.0])
>>> # execute function
>>> assert is_float(var) is True, 'var is a float'
>>> # verify results
>>> print(result)
utool.util_type.is_func_or_method(var)[source]
utool.util_type.is_func_or_method_or_partial(var)[source]
utool.util_type.is_funclike(var)[source]
utool.util_type.is_int(var)[source]
Returns:True if var is an integer.
Return type:bool

Note

Yuck, isinstance(True, int) returns True. This function does not have that flaw.

References

http://www.peterbe.com/plog/bool-is-int

CommandLine:

python -m utool.util_type --test-is_int

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_type import *  # NOQA
>>> var1 = 1
>>> var2 = np.array([1, 2, 3])
>>> var3 = True
>>> var4 = np.array([True, True, False])
>>> result = [is_int(var) for var in [var1, var2, var3, var4]]
>>> print(result)
[True, True, False, False]
utool.util_type.is_list(var)[source]
utool.util_type.is_listlike(var)[source]
utool.util_type.is_method(var)[source]
utool.util_type.is_str(var)[source]
utool.util_type.is_tuple(var)[source]
utool.util_type.is_type(var, valid_types)[source]

Checks for types accounting for numpy

utool.util_type.is_valid_floattype(type_)[source]
Parameters:type (type) – type to check
Returns:if a type_ is a valid float type_ (not variable)
Return type:bool
utool.util_type.smart_cast(var, type_)[source]

casts var to type, and tries to be clever when var is a string

Parameters:
  • var (object) – variable to cast
  • type (type or str) – type to attempt to cast to
Returns:
Return type:

object

CommandLine:

python -m utool.util_type --exec-smart_cast

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_type import *  # NOQA
>>> var = '1'
>>> type_ = 'fuzzy_subset'
>>> result = smart_cast(var, type_)
>>> print(result)
[1]
utool.util_type.smart_cast2(var)[source]

if the variable is a string tries to cast it to a reasonable value. maybe can just use eval. FIXME: funcname

Parameters:var (unknown) –
Returns:some var
Return type:unknown

CommandLine:

python -m utool.util_type --test-smart_cast2

Example

>>> # ENABLE_DOCTEST
>>> from utool.util_type import *  # NOQA
>>> import utool as ut
>>> # build test data
>>> var_list = ['?', 1, '1', '1.0', '1.2', 'True', None, 'None']
>>> # execute function
>>> castvar_list = [smart_cast2(var) for var in var_list]
>>> # verify results
>>> result = ut.list_str(castvar_list, nl=False)
>>> print(result)
['?', 1, 1, 1.0, 1.2, True, None, None]
utool.util_type.try_cast(var, type_, default=None)[source]
utool.util_type.type_str(type_)[source]

utool.util_ubuntu module

utool.util_ubuntu.add_new_mimetype_association(ext, mime_name, exe_fpath=None, dry=True)[source]

TODO: move to external manager and generalize

Parameters:
  • ext (str) – extension to associate
  • mime_name (str) – the name of the mime_name to create (defaults to ext)
  • exe_fpath (str) – executable location if this is for one specific file

References

https://wiki.archlinux.org/index.php/Default_applications#Custom_file_associations

Parameters:
  • ext (str) – extension to associate
  • exe_fpath (str) – executable location
  • mime_name (str) – the name of the mime_name to create (defaults to ext)

CommandLine:

python -m utool.util_ubuntu --exec-add_new_mimetype_association
# Add ability to open ipython notebooks via double click
python -m utool.util_ubuntu --exec-add_new_mimetype_association --mime-name=ipynb+json --ext=.ipynb --exe-fpath=/usr/local/bin/ipynb
python -m utool.util_ubuntu --exec-add_new_mimetype_association --mime-name=ipynb+json --ext=.ipynb --exe-fpath=jupyter-notebook --force

Example

>>> # SCRIPT
>>> from utool.util_ubuntu import *  # NOQA
>>> ext = ut.get_argval('--ext', type_=str, default=None)
>>> mime_name = ut.get_argval('--mime_name', type_=str, default=None)
>>> exe_fpath = ut.get_argval('--exe_fpath', type_=str, default=None)
>>> dry = not ut.get_argflag('--force')
>>> result = add_new_mimetype_association(ext, mime_name, exe_fpath, dry)
>>> print(result)
utool.util_ubuntu.make_application_icon(exe_fpath, dry=True, props={})[source]

CommandLine:

python -m utool.util_ubuntu --exec-make_application_icon --exe=cockatrice --icon=/home/joncrall/code/Cockatrice/cockatrice/resources/cockatrice.png
python -m utool.util_ubuntu --exec-make_application_icon --exe=/opt/zotero/zotero --icon=/opt/zotero/chrome/icons/default/main-window.ico

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_ubuntu import *  # NOQA
>>> import utool as ut
>>> exe_fpath = ut.get_argval('--exe', default='cockatrice')
>>> icon = ut.get_argval('--icon', default=None)
>>> dry = not ut.get_argflag(('--write', '-w'))
>>> props = {'terminal': False, 'icon': icon}
>>> result = make_application_icon(exe_fpath, dry, props)
>>> print(result)

utool.util_web module

utool.util_web.get_localhost()[source]
utool.util_web.is_local_port_open(port)[source]

References

http://stackoverflow.com/questions/7436801/identifying-listening-ports-using-python

utool.util_web.start_simple_webserver(domain=None, port=5832)[source]

simple webserver that echos its arguments

Parameters:
  • domain (None) – (default = None)
  • port (int) – (default = 5832)

CommandLine:

python -m utool.util_web --exec-start_simple_webserver:0
python -m utool.util_web --exec-start_simple_webserver:1

Example

>>> # DISABLE_DOCTEST
>>> from utool.util_web import *  # NOQA
>>> domain = None
>>> port = 5832
>>> result = start_simple_webserver(domain, port)
>>> print(result)

utool.util_win32 module

utool.util_win32.add_to_win32_PATH(script_fpath, *add_path_list)[source]

Writes a registery script to update the PATH variable into the sync registry

CommandLine:

python -m utool.util_win32 --test-add_to_win32_PATH --newpath "C:\Program Files (x86)\Graphviz2.38\bin"

Example

>>> # SCRIPT
>>> from utool.util_win32 import *  # NOQA
>>> script_fpath = join(ut.truepath('~'), 'Sync/win7/registry', 'UPDATE_PATH.reg')
>>> new_path = ut.get_argval('--newpath', str, default=None)
>>> result = add_to_win32_PATH(script_fpath, new_path)
>>> print(result)
utool.util_win32.get_regstr(regtype, var, val)[source]
utool.util_win32.make_regfile_str(key, varval_list, rtype)[source]

Module contents

UTool - Useful Utility Tools
Your friendly neighborhood utility tools

TODO: INSERT APACHE VERSION 2.0 LICENCE IN ALL FILES (Although it should be implied that the entire module and repo is released under that licence.)

pip install git+https://github.com/Erotemic/utool.git@next sudo pip install git+https://github.com/tpircher/quine-mccluskey.git

utool.PROGRESS_FLUSH()

flush() -> None. Flush the internal I/O buffer.

utool.reassign_submodule_attributes(verbose=True)[source]

why reloading all the modules doesnt do this I don’t know

utool.reload_subs(verbose=True)[source]

Reloads utool and submodules

utool.rrrr(verbose=True)

Reloads utool and submodules