line_profiler package

Subpackages

Submodules

Module contents

Line Profiler

The line_profiler module for doing line-by-line profiling of functions

Github

https://github.com/pyutils/line_profiler

Pypi

https://pypi.org/project/line_profiler

ReadTheDocs

https://kernprof.readthedocs.io/en/latest/

Installation

Releases of line_profiler and kernprof can be installed using pip

pip install line_profiler

The package also provides extras for optional dependencies, which can be installed via:

pip install line_profiler[all]

Line Profiler Basic Usage

To demonstrate line profiling, we first need to generate a Python script to profile. Write the following code to a file called demo_primes.py:

from line_profiler import profile


@profile
def is_prime(n):
    '''
    Check if the number "n" is prime, with n > 1.

    Returns a boolean, True if n is prime.
    '''
    max_val = n ** 0.5
    stop = int(max_val + 1)
    for i in range(2, stop):
        if n % i == 0:
            return False
    return True


@profile
def find_primes(size):
    primes = []
    for n in range(size):
        flag = is_prime(n)
        if flag:
            primes.append(n)
    return primes


@profile
def main():
    print('start calculating')
    primes = find_primes(100000)
    print(f'done calculating. Found {len(primes)} primes.')


if __name__ == '__main__':
    main()

In this script we explicitly import the @profile function from line_profiler, and then we decorate function of interest with @profile.

By default nothing is profiled when running the script.

python demo_primes.py

The output will be

start calculating
done calculating. Found 9594 primes.

The quickest way to enable profiling is to set the environment variable LINE_PROFILE=1 and running your script as normal.

LINE_PROFILE=1 python demo_primes.py

This will output 3 files: profile_output.txt, profile_output_<timestamp>.txt, and profile_output.lprof; and stdout will look something like:

start calculating
done calculating. Found 9594 primes.
Timer unit: 1e-09 s

  0.65 seconds - demo_primes.py:4 - is_prime
  1.47 seconds - demo_primes.py:19 - find_primes
  1.51 seconds - demo_primes.py:29 - main
Wrote profile results to profile_output.txt
Wrote profile results to profile_output_2023-08-12T193302.txt
Wrote profile results to profile_output.lprof
To view details run:
python -m line_profiler -rtmz profile_output.lprof

The details contained in the output txt files or by running the script provided in the output will show detailed line-by-line timing information for each decorated function.

Timer unit: 1e-06 s

Total time: 0.731624 s
File: ./demo_primes.py
Function: is_prime at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     4                                           @profile
     5                                           def is_prime(n):
     6                                               '''
     7                                               Check if the number "n" is prime, with n > 1.
     8
     9                                               Returns a boolean, True if n is prime.
    10                                               '''
    11    100000      14178.0      0.1      1.9      max_val = n ** 0.5
    12    100000      22830.7      0.2      3.1      stop = int(max_val + 1)
    13   2755287     313514.1      0.1     42.9      for i in range(2, stop):
    14   2745693     368716.6      0.1     50.4          if n % i == 0:
    15     90406      11462.9      0.1      1.6              return False
    16      9594        922.0      0.1      0.1      return True


Total time: 1.56771 s
File: ./demo_primes.py
Function: find_primes at line 19

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    19                                           @profile
    20                                           def find_primes(size):
    21         1          0.2      0.2      0.0      primes = []
    22    100001      10280.4      0.1      0.7      for n in range(size):
    23    100000    1544196.6     15.4     98.5          flag = is_prime(n)
    24    100000      11375.4      0.1      0.7          if flag:
    25      9594       1853.2      0.2      0.1              primes.append(n)
    26         1          0.1      0.1      0.0      return primes


Total time: 1.60483 s
File: ./demo_primes.py
Function: main at line 29

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    29                                           @profile
    30                                           def main():
    31         1         14.0     14.0      0.0      print('start calculating')
    32         1    1604795.1    2e+06    100.0      primes = find_primes(100000)
    33         1         20.6     20.6      0.0      print(f'done calculating. Found {len(primes)} primes.')

See also

Limitations

Line profiling does have limitations, and it is important to be aware of them. Profiling multi-threaded, multi-processing, and asynchronous code may produce unexpected or no results. All profiling also adds some amount of overhead to the runtime, which may influence which parts of the code become bottlenecks.

Line profiler only measures the time between the start and end of a Python call, so for benchmarking GPU code (e.g. with torch), which have asynchronous or delayed behavior, it will only show the time to sync blocking calls in the main thread.

Other profilers have different limitations and different trade-offs. It’s good to be aware of the right tool for the job. Here is a short list of other profiling tools:

  • Scalene: A CPU+GPU+memory sampling based profiler.

  • PyInstrument: A call stack profiler.

  • Yappi: A tracing profiler that is multithreading, asyncio and gevent aware.

  • profile / cProfile: The builtin profile module.

  • timeit: The builtin timeit module for profiling single statements.

  • timerit: A multi-statements alternative to the builtin timeit module.

  • torch.profiler tools for profiling torch code.

class line_profiler.LineProfiler[source]

Bases: LineProfiler, ByCountProfilerMixin

A profiler that records the execution times of individual lines.

This provides the core line-profiler functionality.

Example

>>> import line_profiler
>>> profile = line_profiler.LineProfiler()
>>> @profile
... def func():
...     x1 = list(range(10))
...     x2 = list(range(100))
...     x3 = list(range(1000))
>>> func()
>>> profile.print_stats()
__call__(func: Callable) → Callable[source]

Decorate a function, method, property, partial() object etc. to start the profiler on function entry and stop it on function exit.

wrap_callable(func: Callable) → Callable[source]
add_callable(func: object, guard: Callable[[Callable], bool] | None = None, name: str | None = None) → Literal[0, 1][source]

Register a function, method, property, partial() object, etc. with the underlying Cython profiler.

Parameters:

  • func (…) – Function, class/static/bound method, property, etc.

  • guard (Optional[Callable[[Callable], bool]]) – Optional checker callable, which takes a function object and returns true(-y) if it should not be passed to add_function(). Defaults to checking whether the function is already a profiling wrapper.

  • name (Optional[str]) – Optional name for func, to be used in log messages.

Returns:

1 if any function is added to the profiler, 0 otherwise.

Note

This method should in general be called instead of the more low-level add_function().

get_stats() → LineStats[source]
dump_stats(filename: PathLike[str] | str) → None[source]

Dump a representation of the data to a file as a pickled LineStats object from get_stats().

print_stats(stream: TextIOBase | None = None, output_unit: float | None = None, stripzeros: bool = False, details: bool = True, summarize: bool = False, sort: bool = False, rich: bool = False, *, config: str | PathLike[str] | bool | None = None) → None[source]

Show the gathered statistics.

add_class(cls: type, *, scoping_policy: ScopingPolicy | str | ScopingPolicyDict | None = None, wrap: bool = False) → int[source]

Add the members (callables (wrappers), methods, classes, …) in a class’ local namespace and profile them.

Parameters:

  • cls (type) – Class to be profiled.

  • scoping_policy (Union[str, ScopingPolicy, ScopingPolicyDict, None]) – Whether (and how) to match the scope of members and decide on whether to add them:

    str (incl. ScopingPolicy):

    Strings are converted to ScopingPolicy instances in a case-insensitive manner, and the same policy applies to all members.

    {'func': ..., 'class': ..., 'module': ...}

    Mapping specifying individual policies to be enacted for the corresponding member types.

    None

    The default, equivalent to DEFAULT_SCOPING_POLICIES.

    See ScopingPolicy and ScopingPolicy.to_policies() for details.

  • wrap (bool) – Whether to replace the wrapped members with wrappers which automatically enable/disable the profiler when called.

Returns:

Number of members added to the profiler.

Return type:

n (int)

add_module(mod: ModuleType, *, scoping_policy: ScopingPolicy | str | ScopingPolicyDict | None = None, wrap: bool = False) → int[source]

Add the members (callables (wrappers), methods, classes, …) in a module’s local namespace and profile them.

Parameters:

  • mod (ModuleType) – Module to be profiled.

  • scoping_policy (Union[str, ScopingPolicy, ScopingPolicyDict, None]) – Whether (and how) to match the scope of members and decide on whether to add them:

    str (incl. ScopingPolicy):

    Strings are converted to ScopingPolicy instances in a case-insensitive manner, and the same policy applies to all members.

    {'func': ..., 'class': ..., 'module': ...}

    Mapping specifying individual policies to be enacted for the corresponding member types.

    None

    The default, equivalent to DEFAULT_SCOPING_POLICIES.

    See ScopingPolicy and ScopingPolicy.to_policies() for details.

  • wrap (bool) – Whether to replace the wrapped members with wrappers which automatically enable/disable the profiler when called.

Returns:

Number of members added to the profiler.

Return type:

n (int)

class line_profiler.LineStats(timings: _TimingsMap, unit: float)[source]

Bases: LineStats

timings: _TimingsMap
unit: float
print(stream: TextIOBase | None = None, output_unit: float | None = None, stripzeros: bool = False, details: bool = True, summarize: bool = False, sort: bool = False, rich: bool = False, *, config: str | PathLike[str] | bool | None = None) → None[source]
to_file(filename: PathLike[str] | str) → None[source]

Pickle the instance to the given filename.

classmethod from_files(file: PathLike[str] | str, /, *files: PathLike[str] | str) → Self[source]

Utility function to load an instance from the given filenames.

classmethod from_stats_objects(stats: _StatsLike, /, *more_stats: _StatsLike) → Self[source]

Example

>>> stats1 = LineStats(
...     {('foo', 1, 'spam.py'): [(2, 10, 300)],
...      ('bar', 10, 'spam.py'):
...      [(11, 2, 1000), (12, 1, 500)]},
...     1E-6)
>>> stats2 = LineStats(
...     {('bar', 10, 'spam.py'):
...      [(11, 10, 20000), (12, 5, 1000)],
...      ('baz', 5, 'eggs.py'): [(5, 2, 5000)]},
...     1E-7)
>>> stats_combined = LineStats.from_stats_objects(
...     stats1, stats2)
>>> assert stats_combined.unit == 1E-6
>>> assert stats_combined.timings == {
...     ('foo', 1, 'spam.py'): [(2, 10, 300)],
...     ('bar', 10, 'spam.py'):
...     [(11, 12, 3000), (12, 6, 600)],
...     ('baz', 5, 'eggs.py'): [(5, 2, 500)]}
line_profiler.load_ipython_extension(ip: object) → None[source]

API for IPython to recognize this module as an IPython extension.

line_profiler.load_stats(file: PathLike[str] | str, /, *files: PathLike[str] | str) → Self

Utility function to load an instance from the given filenames.

line_profiler.main() → None[source]

The line profiler CLI to view output from kernprof -l.

line_profiler.show_func(filename: str, start_lineno: int, func_name: str, timings: Sequence[tuple[int, int, int | float]], unit: float, output_unit: float | None = None, stream: TextIOBase | None = None, stripzeros: bool = False, rich: bool = False, *, config: str | PathLike[str] | bool | None = None) → None[source]

Show results for a single function.

Parameters:

  • filename (str) – Path to the profiled file

  • start_lineno (int) – First line number of profiled function

  • func_name (str) – name of profiled function

  • timings (List[Tuple[int, int, float]]) – Measurements for each line (lineno, nhits, time).

  • unit (float) – The number of seconds used as the cython LineProfiler’s unit.

  • output_unit (float | None) – Output unit (in seconds) in which the timing info is displayed.

  • stream (io.TextIOBase | None) – Defaults to sys.stdout

  • stripzeros (bool) – If True, prints nothing if the function was not run

  • rich (bool) – If True, attempt to use rich highlighting.

  • config (Union[str, PurePath, bool, None]) – Optional filename from which to load configurations (e.g. output column widths); default (= True or None) is to look for a config file based on the environment variable ${LINE_PROFILER_RC} and path-based lookup; passing False disables all lookup and falls back to the default configuration

Example

>>> from line_profiler.line_profiler import show_func
>>> import line_profiler
>>> # Use a function in this file as an example
>>> func = line_profiler.line_profiler.show_text
>>> start_lineno = func.__code__.co_firstlineno
>>> filename = func.__code__.co_filename
>>> func_name = func.__name__
>>> # Build fake timeings for each line in the example function
>>> import inspect
>>> num_lines = len(inspect.getsourcelines(func)[0])
>>> line_numbers = list(range(start_lineno + 3,
...                           start_lineno + num_lines))
>>> timings = [(lineno, idx * 1e13, idx * (2e10 ** (idx % 3)))
...            for idx, lineno
...            in enumerate(line_numbers, start=1)]
>>> unit = 1.0
>>> output_unit = 1.0
>>> stream = None
>>> stripzeros = False
>>> rich = 1
>>> show_func(filename, start_lineno, func_name, timings, unit,
...           output_unit, stream, stripzeros, rich)
line_profiler.show_text(stats: _TimingsMap, unit: float, output_unit: float | None = None, stream: io.TextIOBase | None = None, stripzeros: bool = False, details: bool = True, summarize: bool = False, sort: bool = False, rich: bool = False, *, config: str | PathLike[str] | bool | None = None) → None[source]

Show text for the given timings.