7 fIddlZddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZmZmZmZmZmZmZmZmZerddlmZmZGd d ej ej!e"Z#Gd d e#e"Z$Gd de#e"Z%dZ&ej'dddkr"e&dz Z&ej'dddkr e&dz Z&e&dz Z&ddZ(dS)N)Module)IterableIteratorListOptionalTupleAnycast TYPE_CHECKING)xrange) LineNumbers) Token match_tokenis_non_coding_tokenpatched_generate_tokens last_stmtannotate_fstring_nodesgenerate_tokens is_moduleis_stmt)AstNode TokenInfocDeZdZdZejdZddZddZdS) ASTTextBasecr||_tj|}||_t ||_dSN) _filenamesix ensure_text_textr _line_numbers)self source_textfilenames I/var/www/equiseq/venv/lib/python3.11/site-packages/asttokens/asttokens.py__init__zASTTextBase.__init__%s9 DN /+..KDJ$[11Dct) Returns two ``(lineno, col_offset)`` tuples for the start and end of the given node. If the positions can't be determined, or the nodes don't correspond to any particular text, returns ``(1, 0)`` for both. ``padded`` corresponds to the ``padded`` argument to ``ast.get_source_segment()``. This means that if ``padded`` is True, the start position will be adjusted to include leading whitespace if ``node`` is a multiline statement. )NotImplementedErrorr#nodepaddeds r&get_text_positionszASTTextBase.get_text_positions4s  r(Tcp|||\}}|jj||jj|fS)a Returns the (startpos, endpos) positions in source text corresponding to the given node. Returns (0, 0) for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r/r"line_to_offsetr#r-r.startends r&get_text_rangezASTTextBase.get_text_rangeBsG((v66JE3'd'/'d'- r(cR|||\}}|j||S)z Returns the text corresponding to the given node. Returns '' for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r5r!r2s r&get_textzASTTextBase.get_textPs.$$T622JE3 :eSj !!r(N)T) __name__ __module__ __qualname__r'abcabstractmethodr/r5r7r(r&rr$sj 2 2 2        " " " " " "r(rceZdZdZdfd ZdZdZedZed Z ed Z ed Z d Z d Z dZddZddZddZ ddZddZdZxZS) ASTTokensa ASTTokens maintains the text of Python code in several forms: as a string, as line numbers, and as tokens, and is used to mark and access token and position information. ``source_text`` must be a unicode or UTF8-encoded string. If you pass in UTF8 bytes, remember that all offsets you'll get are to the unicode text, which is available as the ``.text`` property. If ``parse`` is set, the ``source_text`` will be parsed with ``ast.parse()``, and the resulting tree marked with token info and made available as the ``.tree`` property. If ``tree`` is given, it will be marked and made available as the ``.tree`` property. In addition to the trees produced by the ``ast`` module, ASTTokens will also mark trees produced using ``astroid`` library . If only ``source_text`` is given, you may use ``.mark_tokens(tree)`` to mark the nodes of an AST tree created separately. FN ctt||||rtj||n||_|t |j}t| ||_ d|j D|_ |jr| |jdSdS)Ncg|] }|j Sr=)startpos).0toks r& z&ASTTokens.__init__..s@@@C3<@@@r() superr?r'astparse_treerr!list_translate_tokens_tokens_token_offsets mark_tokens)r#r$rItreer%tokens __class__s r&r'zASTTokens.__init__ps  )T##K:::5:D;111DJ~tz**f..v6677DLA@4<@@@D z# tz"""""##r(cNddlm}|||dS)ap Given the root of the AST or Astroid tree produced from source_text, visits all nodes marking them with token and position information by adding ``.first_token`` and ``.last_token``attributes. This is done automatically in the constructor when ``parse`` or ``tree`` arguments are set, but may be used manually with a separate AST or Astroid tree. r ) MarkTokensN)rOrT visit_tree)r# root_noderTs r&rOzASTTokens.mark_tokenss:('''''Jt *****r(c#"Ktt|D]o\}}|\}}}}}t|||||||j|d|d|j|d|dVpdS)zS Translates the given standard library tokens into our own representation. rr N) enumeraterrr"r1) r#original_tokensindexrEtok_typetok_strr3r4lines r&rLzASTTokens._translate_tokenss  7 H HIIEE s,/)hT (GUCu$33E!HeAhGG$33CFCFCC E EEEEEEEr(c|jS)z,The source code passed into the constructor.)r!r#s r&textzASTTokens.text  :r(c|jS)zIThe list of tokens corresponding to the source code from the constructor.)rMr_s r&rQzASTTokens.tokenss  <r(c|jS)zTThe root of the AST tree passed into the constructor or parsed from the source code.)rJr_s r&rPzASTTokens.treerar(c|jS)zThe filename that was parsed)rr_s r&r%zASTTokens.filenames  >r(cR|jtj|j|dz S)z Returns the token containing the given character offset (0-based position in source text), or the preceeding token if the position is between tokens. r )rMbisectrN)r#offsets r&get_token_from_offsetzASTTokens.get_token_from_offsets% < d&96BBQF GGr(c^||j||S)z Returns the token containing the given (lineno, col_offset) position, or the preceeding token if the position is between tokens. )rhr"r1r#lineno col_offsets r& get_tokenzASTTokens.get_tokens,  % %d&8&G&GPZ&[&[ \ \\r(c`|||j||S)zd Same as get_token(), but interprets col_offset as a UTF8 offset, which is what `ast` uses. )rmr" from_utf8_colrjs r&get_token_from_utf8zASTTokens.get_token_from_utf8s+ >>&$"4"B"B6:"V"V W WWr(c|jdz}|sCt|j|jr$|dz }t|j|j$|j|S)z Returns the next token after the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r rZrrMtyper#rE include_extrais r& next_tokenzASTTokens.next_token` A A   Q 4 5 5 Q  Q 4 5 5 <?r(c|jdz }|sCt|j|jr$|dz}t|j|j$|j|S)z Returns the previous token before the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r rrrts r& prev_tokenzASTTokens.prev_tokenrxr(c|}|r|jn|j}t|||sPtj|js7||d}t|||stj|j7|S)z Looks for the first token, starting at start_token, that matches tok_type and, if given, the token string. Searches backwards if reverse is True. Returns ENDMARKER token if not found (you can check it with `token.ISEOF(t.type)`). Tru)rzrwrtokenISEOFrs)r# start_tokenr[r\reversetadvances r& find_tokenzASTTokens.find_tokens A!(=doodoG!Xw//) AF8K8K) '!4 ( ( (a!Xw//) AF8K8K) Hr(c#Kt|j|jdzD]2}|st|j|js|j|V3dS)z Yields all tokens in order from first_token through and including last_token. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. r N)r rZrrMrs)r# first_token last_tokenrurvs r& token_rangezASTTokens.token_rangesjK%z'7!'; < < 1$,q/2FGGl1or(cF||j|j|S)z Yields all tokens making up the given node. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. r|)rrr)r#r-rus r& get_tokenszASTTokens.get_tokenss$   D,do]  [ [[r(ct|dsdS|jj}|jj}|r6t d||Dr |ddf}||fS)r*rr rrc3JK|]}t|tjVdSr)rr}NEWLINE)rDrs r& z/ASTTokens.get_text_positions..s.SSk!U]33SSSSSSr(r)hasattrrr3rr4anyrr2s r&r/zASTTokens.get_text_positionssy 4 ' ' ^   "E / C #SST__T=R=RSSSSSQxme #:r()FNr@N)F)NF)r8r9r:__doc__r'rOrLpropertyr`rQrPr%rhrmrprwrzrrrr/ __classcell__rRs@r&r?r?\s&######( + + + E E E  8   8   8   8 HHH ] ] ]XXX             !&    \\\\r(r?c\eZdZdZd fd ZedZedZdZdZ xZ S) ASTTextaF Supports the same ``get_text*`` methods as ``ASTTokens``, but uses the AST to determine the text positions instead of tokens. This is faster than ``ASTTokens`` as it requires less setup work. It also (sometimes) supports nodes inside f-strings, which ``ASTTokens`` doesn't. Some node types and/or Python versions are not supported. In these cases the ``get_text*`` methods will fall back to using ``ASTTokens`` which incurs the usual setup cost the first time. If you want to avoid this, check ``supports_tokenless(node)`` before calling ``get_text*`` methods. Nr@ctt|||||_|jt |jd|_dSr)rGrr'rJr _asttokens)r#r$rPr%rRs r&r'zASTText.__init__,sO  '4!!+x888DJ zTZ(((DOOOr(c|j8tj|j|j|_t |j|jSr)rJrHrIr!rrr_s r&rPz ASTText.tree9s; z9TZ88djTZ((( :r(cj|j&t|j|j|j|_|jS)N)rPr%)rr?r!rPrr_s r& asttokenszASTText.asttokensAs; ! *y>do ?r(ctjdddkrtdt|r.d|jt |jfSt|dddS|sJt|dg}|s"t|d d}t|d g}|r |d }n|}|j }t|}|r9||j ks+||j kr#t|d drt|rd }n |j ||j}||f} tt |j } tt |j} |j | | } | | f} | | fS) zF Version of ``get_text_positions()`` that doesn't use tokens. NzPThis method should only be called internally after checking supports_tokenless()rrkrdecorator_list decoratorsnodesrdoc_node)sys version_infoAssertionErrorrr"offset_to_linelenr!getattrrkr end_linenorrorlr intend_col_offset) r#r-r.rdecorators_node start_node start_linenoend_nodestart_col_offsetr3rrr4s r&_get_text_positions_tokenlessz%ASTText._get_text_positions_tokenlessLs  f$$ m n nnH T'66s4:GG GGtXt$$, ^ KK4/44J 9lD99o?GR88ja=jjj$LH_'' DO + +j$// ,  , +99, H]^^ + ,E c8.//J#x677N'55j.QQN ~ &C #:r(ct|ddrdSt|r|||S|j||S)r*_broken_positionsNr)rsupports_tokenlessrrr/r,s r&r/zASTText.get_text_positionss\t($// ^$>  / /f = == > , ,T6 : ::r()Nr@) r8r9r:rr'rrPrrr/rrs@r&rrs          8  8DDDL;;;;;;;r(rr=rr) arguments Argumentswithitem)argStarred)SliceExtSliceIndexkeywordcNt|jtvot|tj o6|duo2t|jdkot|jjdv o7tjdddkodtj vS)a Returns True if the Python version and the node (if given) are supported by the ``get_text*`` methods of ``ASTText`` without falling back to ``ASTTokens``. See ``ASTText`` for why this matters. The following cases are not supported: - Python 3.7 and earlier - PyPy - ``ast.arguments`` / ``astroid.Arguments`` - ``ast.withitem`` - ``astroid.Comprehension`` - ``astroid.AssignName`` inside ``astroid.Arguments`` or ``astroid.ExceptHandler`` - The following nodes in Python 3.8 only: - ``ast.arg`` - ``ast.Starred`` - ``ast.Slice`` - ``ast.ExtSlice`` - ``ast.Index`` - ``ast.keyword`` N AssignName)r ExceptHandlerrrpypy) rsr8_unsupported_tokenless_types isinstancerHASTparentrrversionlower)r-s r&rrs0 4jj!== , tSW % %% $d*: JJ < / MT[!!*.LL   ,  2A2 & ( ,  ))++ + r(r))r;rHrfrr}rtypingrrrrrr r r r six.movesr line_numbersrutilrrrrrrrrrrrwith_metaclassABCMetaobjectrr?rrrrr=r(r&rs^ VVVVVVVVVVVVVVVVVVVV %%%%%% '&&&&&&&&5"5"5"5"5"$#$S[&995"5"5"p@@@@@ V@@@FE;E;E;E;E;k6E;E;E;R "BQB6!!"HHbqbV## $66  $MM $$$$$$r(