Coverage for main.py: 85%
Shortcuts on this page
r m x toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
1# -*- coding: utf-8 -*-
2# python-todo-to-issue/main.py
4# Copyright (c) 2021, Kevin Sawade (kevin.sawade@uni-konstanz.de)
5# All rights reserved.
6#
7# Redistribution and use in source and binary forms, with or without
8# modification, are permitted provided that the following conditions are met:
9#
10# * Redistributions of source code must retain the above copyright
11# notice, this list of conditions and the following disclaimer.
12# * Redistributions in binary form must reproduce the above copyright
13# notice, this list of conditions and the following disclaimer in the
14# documentation and/or other materials provided with the distribution.
15# * Neither the name of the copyright holders nor the names of any
16# contributors may be used to endorse or promote products derived
17# from this software without specific prior written permission.
18#
19# This file is free software: you can redistribute it and/or modify
20# it under the terms of the GNU Lesser General Public License as
21# published by the Free Software Foundation, either version 2.1
22# of the License, or (at your option) any later version.
23#
24# This file is distributed in the hope that it will be useful,
25# but WITHOUT ANY WARRANTY; without even the implied warranty of
26# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
27# GNU Lesser General Public License for more details.
28#
29# Find the GNU Lesser General Public License under <http://www.gnu.org/licenses/>.
30"""
31Convert python Todos to github issues
32=====================================
34Preface
35-------
37This module converts todos from your project to issues on github. I took lots
38of inspiration from Alastair Mooney's <a href="https://github.com/alstr/todo-to-issue-action">todo-to-issue-action</a>,
39which is a much more complete library and allows more languages than just python.
40However, it does not recognize google-style <a href="https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html#example-google">Todo labels</a>
41, doesn't allow to skip todos (like doctest's skip) and has a proprietary diff
42parser instead of using <a href="https://github.com/matiasb/python-unidiff">unidiff</a>.
44Installation
45------------
47This repo offers a GitHub action, that can be integrated into your GitHub workflows.
48If you are confident with GitHub actions, you can follow the quickstart on this projects <a href="https://github.com/kevinsawade/python-todo-to-issue/blob/main/README.md">README.md</a>.
49page to quickly set up the this action. Otherwise you can follow these instructions:
51- Visit the GitHub marketplace and find the latest version of this action: https://github.com/marketplace/actions/python-todo-to-issue-action.
52- If you want to have a dedicated bot open the issues you have to create a token with the issue scope on your tokens page: https://github.com/settings/tokens
53- Go to the settings of the repo you want to use this action in and add the token as a new repository secret. Give it a descriptive name, like CUSTOM_ISSUE_TOKEN.
54- Create a .yml file at .github/workflows/todo-to-issue.yml with this syntax:
56```yaml
57name: Todo-to-Issue
59on:
60 push:
61 branches: [ main ]
62 pull_request:
63 branches: [ main ]
65jobs:
66 todo-to-issue:
67 # The type of runner that the job will run on
68 runs-on: ubuntu-latest
69 strategy:
70 matrix:
71 python-version: [3.9]
73 steps:
74 - name: Checkout 🛎️
75 uses: actions/checkout@v2
77 - name: Create Issues ✔️
78 uses: kevinsawade/python-todo-to-issue@latest
79 with:
80 # GitHub Bot will open the issues:
81 TOKEN: ${{ secrets.GITHUB_TOKEN }}
82 # Provide a custom secret
83 # TOKEN: ${{ secrets.CUSTOM_ISSUE_TOKEN }}
85```
87All your todos will be converted to issues, once you push to github.
89What is regarded as a Todo?
90---------------------------
92First of all: Only Todos from commits are used as issues. If you have old Todos
93in your module, you need to remove them, commit and then include them again.
95Todos are searched for in comments which start with ``# Todo:``. You can expand
96these comments with the assignee of the issue. For this you put the github
97username of a maintainer, developer or owner of a repo in parentheses. You can
98also write multi-line todos, by indenting them with extra spaces. Using this
99multi-line syntax, you can add labels and milestone to an issue.
101```python
102# todo: This is a simple in-line todo. This will be the title of the issue.
104# todo (kevinsawade): I will fix this weird contraption.
106# Todo: This is the title of a mutli-line issue.
107# This is the body of the multi-line issue. Here, you can specify
108# What needs to be done to fix this issue. Issues are automatically
109# closed, once the Todo is removed from the file. You can set assignees,
110# labels and milestone like so:
111# assignees: kevinsawade, github_user3
112# labels: devel, bug
113# milestone: release
114```
116Besides these in-line Todos, Todos from google-style formatted docstrings will
117also be picked up. The general style is the same. Indentation is done via
1184 spaces. Assignees can be put in parentheses or as an extra line in multi-line Todos.
120```python
121def myfunc(arg1):
122 \"\"\"This is the overview docstring.
124 This is more detailed info to the function `myfunc`.
126 Args:
127 arg1 (str): Argument `arg1` should be of type `str`.
129 Todo:
130 * Single-line Todos are introduced as a single bullet-point.
131 * This line becomes the title of the GitHub issue.
132 * (kevinsawade) Assignees are put into parentheses.
133 * Titles for multi-line Todos are also bullet-points.
134 But the body is indented according to google's styleguide.
135 Assignees, labels and milestones are added similar to the in-line
136 Todos. Once the Todo has been removed from the file, the issue
137 is closed.
138 assignees: kevinsawade, github_user2
139 labels: devel, bug
140 milestone: alpha
142 \"\"\"
143return 'Hello!' + arg1
144```
146Excluding Todos from being turned into issues
147---------------------------------------------
149To skip todos you can add ``todo: +SKIP`` after the todo-line. This one is not
150case insensitive and only works if you use it verbose.
152Todos after code lines
153----------------------
155If you write your Todos after code lines like so:
157```python
159a = str(1) # todo: This line is bad code
160b = '2' # todo (kevinsawade): This line is better.
161```
163You can add these after-code-todos with an additional
164`INCLUDE_TODO_AFTER_CODE_LINE` option to the yaml file:
166```yaml
167- name: Create Issues ✔️
168 uses: kevinsawade/python-todo-to-issue@latest
169 with:
170 TOKEN: ${{ secrets.GITHUB_TOKEN }}
171 INCLUDE_TODO_AFTER_CODE_LINE: ${{ true }}
172```
174Coverage Report
175---------------
177Access the coverage report under: https://kevinsawade.github.io/python-todo-to-issue/htmlcov/index.html
179Classes and Functions
180---------------------
182The remainder of this page contains the functions and classes used to run this
183action. These functions and classes contain their own documentation which might
184help in debugging/ reusing parts of this code.
188"""
189################################################################################
190# Globals
191################################################################################
194# __all__ = ['main', 'GitHubClient']
197################################################################################
198# Regex Patterns
199################################################################################
202TODO_CHARS_PATTERN = '[*#]'
203# Thanks to Alastair Mooney's regexes
205# INLINE_TODO_PATTERN = r'\s*#\s(?i)todo(\:|\s)(\s|\().*'
206INLINE_TODO_PATTERN = r'\s*#\s(?i)todo(\S|\s)(\s|\S|\().*'
207DOCSTRING_TODO_PATTERN = r'\s*\*\s*(\(.*|.*)'
208TODO_SKIP_SUBSTRING = 'todo: +SKIP'
211################################################################################
212# Imports
213################################################################################
216from enum import Enum
217import ast, os, requests, json, git, re, unittest
218from unidiff import PatchSet
219from io import StringIO
220from time import sleep
223################################################################################
224# Functions
225################################################################################
228def join_lines(issue, line_break):
229 """Joins lines using a defined line_break.
231 Args:
232 issue (Issue): An `Issue` instance.
233 line_break (str): The line-break used in `GitHubClient`.
235 Returns:
236 str: A string with the formatted issue body.
238 """
239 annotation = "This issue was automatically created by a github action that converts project Todos to issues."
240 return annotation + '\n\n' + line_break.join(issue.body)
243def get_body(issue, url, line_break):
244 """Constructs a body with issue and url.
246 Args:
247 issue (Issue): An `Issue` instance.
248 url (str): The url constructed from `GitHubClient`.
249 line_break (str): The line-break used in `GitHubClient`.
251 Returns:
252 str: A string with the body of the issue.
254 """
255 formatted_issue_body = join_lines(issue, line_break)
256 formatted = (formatted_issue_body + '\n\n'
257 + url + '\n\n'
258 + '```' + issue.markdown_language + '\n'
259 + issue.hunk + '\n'
260 + '```')
261 return formatted
264def _get_assignees(lines):
265 assignees = []
266 if len(lines) > 1:
267 for i, line in enumerate(lines):
268 if line.lstrip().startswith('assignees:'):
269 lines.pop(i)
270 line = line.lstrip().lstrip('assignees:')
271 assignees = [elem.strip() for elem in line.strip().split(',')]
272 return lines, assignees
273 elif len(lines) == 1:
274 if '(' in lines[0]:
275 s = lines[0]
276 assignees = s[s.find("(") + 1:s.find(")")]
277 lines[0] = lines[0].replace('(' + assignees + ')', '')
278 assignees = [elem.strip() for elem in assignees.strip().split(',')]
279 return lines, assignees
282def _get_labels(lines):
283 labels = []
284 for i, line in enumerate(lines):
285 if line.lstrip().startswith('labels:'):
286 lines.pop(i)
287 line = line.lstrip().lstrip('labels:')
288 labels = [elem.strip() for elem in line.strip().split(',')]
289 return lines, labels
292def _get_milestone(lines):
293 milestone = None
294 for i, line in enumerate(lines):
295 if line.lstrip().startswith(('milestone:', 'milestones:')):
296 lines.pop(i)
297 if 'milestone:' in line:
298 milestone = line.lstrip().lstrip('milestone:').strip()
299 elif 'milestones:' in line:
300 milestone = line.lstrip().lstrip('milestones:').strip()
301 return lines, milestone
304################################################################################
305# Classes
306################################################################################
308class LineStatus(Enum):
309 """Represents the status of a line in a diff file."""
310 ADDED = 0
311 DELETED = 1
312 UNCHANGED = 2
315testing_values = dict(
316 title='TEST AUTO ISSUE',
317 labels=['todo'],
318 assignees=[],
319 milestone=None,
320 body=[
321 'This issue is automatically created by Unittests. If this issue is not automatically closed, tests have failed.'],
322 hunk="# This is the code block that would normally be attached to the issue.\ndef function()\n return 'Hi!'",
323 file_name='main.py',
324 start_line='47',
325 markdown_language='python',
326 status=LineStatus.ADDED
327)
330class Issue:
331 """Issue class, filled with attributes to create issues from.
333 Attributes:
334 title (str): The title of the issue.
335 labels (list[str]): Labels, that should be added to the issue. Default
336 labels contains the 'todo' label.
337 assignees (list[str]): Can be []. Assignees need to be maintainer
338 of the repository.
339 milestone (Union[str, None]): Can be None. Milestone needs to be one of
340 the already defined milestones in the repo.
341 body (list[str]): The lines of the issue body. The issue body will
342 automatically appended with a useful url and a markdown code block
343 defined in `hunk`.
344 hunk (str): Newline separated string of code, that produced the todo.
345 file_name (str): Name of the file that produced the todo. If file is
346 somewhere, the path starting from repo root needs te be included.
347 start_line (str): The line where the todo comment starts.
348 markdown_language (str): The language of `hunk`.
349 status (LineStatus): An instance of the `LineStatus` Enum class.
351 """
353 def __init__(self, testing=0, **kwargs):
354 if testing:
355 for key, val in testing_values.items():
356 self.__setattr__(key, val)
357 return
359 for key, val in kwargs.items():
360 self.__setattr__(key, val)
361 for key in testing_values.keys():
362 try:
363 self.__getattribute__(key)
364 except AttributeError:
365 raise TypeError(f'__init__() missing 1 required argument: {key}')
366 if 'todo' not in self.labels:
367 self.labels.append('todo')
369 def __str__(self):
370 string = f"Title: {self.title}, assignees: [{', '.join(self.assignees)}], milestone: {self.milestone}, status: {self.status}]\n"
371 return string
373 def __repr__(self):
374 return self.__str__()
377class GitHubClient():
378 """Class to interact with GitHub, read and create issues.
380 This class interacts with GitHub via api.github.com and reads issues from
381 repositories. The default behavior is, that it uses git-python to get the
382 url of the current remote origin. This name of the repo is usually built
383 like this: user_name/repo_name or org_name/repo_name. git-python is also
384 used to get the sha of the current commit and the previous commit.
386 About tokens: To work with private repos and to have the ability to close
387 issues in a repository, this class needs github token, sometimes also called
388 a secret. The secret can be created in your github account. Visit
389 https://github.com/settings/tokens and click 'Generate a new token'. Check
390 the 'repo' scope and give the token a descriptive name. If the repo lies
391 within an organisation, the token of a user with access rights to the org
392 repo, will suffice.
394 About secrets: This token should not fall into the wrong hands. However, in
395 production and in testing the token is needed. In production, the token can
396 be provided as a secret. Add the token as a secret in the settings page of
397 your repo. In testing, the token is provided by placing a file called
398 'secrets' in the repo's root directory.
400 Attributes:
401 existing_issues (list): A list of existing issues.
402 testing (bool): Used, when this class is used in testing. Changes some
403 behaviors.
404 repo (str): A string of 'user_name/repo_name' or 'org_name/repo_name'
405 which identifies the repo on github.com.
406 sha (str): The 7 character sha hash of the current commit.
407 before (str): The 7 character sha hash of the previous commit.
408 token (str): A github token.
409 base_url (str): A string containing 'https://api.github.com/'.
410 repos_url (str): base_url + 'repos/'
411 issues_url (str): GitHub API url pointing to a url with the current
412 repo's issues.
413 issue_headers (dict): A dict to provide to requests.get() as header.
415 """
417 def __init__(self, testing=0):
418 """Instantiate the GitHubClient.
420 Keyword Args:
421 testing (bool, optional): Whether class is used during testing.
423 """
424 # set some attributes right from the start
425 self.existing_issues = []
426 self.testing = testing
428 self.repo = os.environ['INPUT_REPO']
429 self.sha = os.environ['INPUT_SHA']
430 self.before = os.environ['INPUT_BEFORE']
432 # get before and current hash
433 if self.testing == 1:
434 self.sha = '036ef2ca'
435 self.before = '11858e41'
436 elif self.testing == 2:
437 self.sha = '7fae83cc'
438 self.before = '036ef2ca'
440 # get the token from environment variables
441 self.token = os.getenv('INPUT_TOKEN')
443 # define line break. Can also be \n\n which formats multi-line todos
444 # nicer.
445 self.line_break = '\n'
447 # set other attributes
448 self.base_url = 'https://api.github.com/'
449 self.repos_url = f'{self.base_url}repos/'
450 self.issues_url = f'{self.repos_url}{self.repo}/issues'
451 self.issue_headers = {
452 'Content-Type': 'application/json',
453 'Authorization': f'token {self.token}'
454 }
456 # get current issues
457 self._get_existing_issues()
459 def get_specific_diff(self, before, after):
460 """Get the diff based on specific commits"""
461 diff_url = f'{self.repos_url}{self.repo}/compare/{before}...{after}'
462 diff_headers = {
463 'Accept': 'application/vnd.github.v3.diff',
464 'Authorization': f'token {self.token}'
465 }
466 diff_request = requests.get(url=diff_url, headers=diff_headers)
467 if diff_request.status_code == 200:
468 return diff_request.text
469 raise Exception('Could not retrieve diff. Operation will abort.')
471 def get_last_diff(self):
472 """Get the last diff based on the SHA of the last two commits."""
473 diff_url = f'{self.repos_url}{self.repo}/compare/{self.before}...{self.sha}'
474 diff_headers = {
475 'Accept': 'application/vnd.github.v3.diff',
476 'Authorization': f'token {self.token}'
477 }
478 diff_request = requests.get(url=diff_url, headers=diff_headers)
479 if diff_request.status_code == 200:
480 return diff_request.text
481 raise Exception('Could not retrieve diff. Operation will abort.')
483 def get_file_at_commit(self, filepath, commit):
484 """Returns a string, with the file contents at the current hashed commit.
486 Args:
487 filepath (str): Path to the file from repo root.
488 commit (str): The short-hash, 7-digit commit.
490 Returns:
491 str: The contents of the file.
493 """
494 raw_file_url = f'{self.repos_url}{self.repo}/contents/{filepath}?ref={commit}'
495 raw_file_headers = {
496 'Accept': 'application/vnd.github.VERSION.raw',
497 'Authorization': f'token {self.token}'
498 }
499 raw_file_request = requests.get(url=raw_file_url, headers=raw_file_headers)
500 if raw_file_request.status_code == 200:
501 return raw_file_request.text
503 def _get_repo_url(self, remote_url):
504 """Construct repo url from ssh or https repo urls"""
505 if '@' not in remote_url:
506 self.repo = remote_url.lstrip('https://github.com/').rstrip('.git')
507 else:
508 self.repo = remote_url.lstrip('git@github.com:').rstrip('.git')
510 def _get_existing_issues(self, page=1):
511 """Populate the existing issues list."""
512 params = {
513 'per_page': 100,
514 'page': page,
515 'state': 'open',
516 'labels': 'todo'
517 }
518 list_issues_request = requests.get(self.issues_url, headers=self.issue_headers, params=params)
519 if list_issues_request.status_code == 200:
520 # check
521 self.existing_issues.extend(list_issues_request.json())
522 links = list_issues_request.links
523 if 'next' in links:
524 self._get_existing_issues(page + 1)
526 @staticmethod
527 def is_same_issue(issue, other_issue, line_break):
528 """Compares two issues.
530 Args:
531 issue (Issue): Instance of `Issue`.
532 other_issue (dict): Json dict returned from GitHub API of another issue.
533 url_to_line (str): The url built by the `GitHubClient`.
534 line_break (str): The line-break used in `GitHubClient`.
536 Returns
537 bool: Whether issues are identical or not.
539 """
540 # check title
541 a = issue.title == other_issue['title']
542 if not 'https://github.com/' in other_issue['body']:
543 return a
544 else:
545 # check issue text
546 this_text = join_lines(issue, line_break).rstrip()
547 other_text = other_issue['body'].split('https://github.com')[0].rstrip()
548 b = this_text == other_text
550 return a and b
552 def create_issue(self, issue):
553 """Creates issue on github from an Issue class.
555 Keyword Args:
556 issue (Issue): An instance of the `Issue` class in this document.
558 """
559 # truncate issue title if too long
560 title = issue.title
561 if len(title) > 80:
562 title = title[:80] + '...'
564 # define url to line
565 url_to_line = f'https://github.com/{self.repo}/blob/{self.sha}/{issue.file_name}#L{issue.start_line}'
567 # construct the issue body
568 body = get_body(issue, url_to_line, self.line_break)
570 # Alastair Mooney's has problems with rebasing. Let's see how this works out
571 # One could use GitHub's GraphQL API
572 for existing_issue in self.existing_issues:
573 if self.__class__.is_same_issue(issue, existing_issue, self.line_break):
574 # The issue_id matching means the issue issues are identical.
575 print(f'Skipping issue (already exists)')
576 return
578 new_issue_body = {'title': title, 'body': body, 'labels': issue.labels}
580 # check whether milestones are existent
581 if issue.milestone:
582 milestone_url = f'{self.repos_url}{self.repo}/milestones/{issue.milestone}'
583 milestone_request = requests.get(url=milestone_url, headers=self.issue_headers)
584 if milestone_request.status_code == 200:
585 new_issue_body['milestone'] = issue.milestone
586 else:
587 print(f'Milestone {issue.milestone} does not exist! Dropping this parameter!')
589 # check whether label exists
590 valid_assignees = []
591 for assignee in issue.assignees:
592 assignee_url = f'{self.repos_url}{self.repo}/assignees/{assignee}'
593 assignee_request = requests.get(url=assignee_url, headers=self.issue_headers)
594 if assignee_request.status_code == 204:
595 valid_assignees.append(assignee)
596 else:
597 print(f'Assignee {assignee} does not exist! Dropping this assignee!')
598 new_issue_body['assignees'] = valid_assignees
600 new_issue_request = requests.post(url=self.issues_url, headers=self.issue_headers,
601 data=json.dumps(new_issue_body))
603 return new_issue_request.status_code
605 def close_issue(self, issue):
606 """Check to see if this issue can be found on GitHub and if so close it.
608 Keyword Args:
609 issue (Issue): An instance of the `Issue` class in this document.
611 """
612 matched = 0
613 issue_number = None
614 for existing_issue in self.existing_issues:
615 # This is admittedly a simple check that may not work in complex scenarios, but we can't deal with them yet.
616 if self.__class__.is_same_issue(issue, existing_issue, self.line_break):
617 print("matched")
618 matched += 1
619 # If there are multiple issues with similar titles, don't try and close any.
620 if matched > 1:
621 print(f'Skipping issue (multiple matches)')
622 break
623 issue_number = existing_issue['number']
624 else:
625 if matched == 0 and self.testing:
626 raise Exception(f"Couldn't match issue {issue.title}, {issue.body}")
627 else:
628 # The titles match, so we will try and close the issue.
629 update_issue_url = f'{self.repos_url}{self.repo}/issues/{issue_number}'
630 body = {'state': 'closed'}
631 close_issue_request = requests.patch(update_issue_url, headers=self.issue_headers, data=json.dumps(body))
633 issue_comment_url = f'{self.repos_url}{self.repo}/issues/{issue_number}/comments'
634 body = {'body': f'Closed in {self.sha}'}
635 update_issue_request = requests.post(issue_comment_url, headers=self.issue_headers,
636 data=json.dumps(body))
637 return update_issue_request.status_code, close_issue_request.status_code
638 return
641class ToDo:
642 """Class that parses google-style docstring todos from git diff hunks.
644 Attributes:
645 line (unidiff.Line): The line that triggered this todo.
646 block (list[str]): The lines following the title of multi-line
647 todos. Every line is one string in this list of str.
648 status (LineStatus): The status of the ToDo. Can be ADDED or DELETED>
649 markdown_language (str): What markdown language to use. Defaults to
650 'python'.
651 hunk (unidiff.Hunk): The hunk that contains the line. Will be converted
652 to code block in the issue.
653 file_name (str): The path of the file from which the todo was extracted.
654 target_line (Union[str, int]): The line number from which the
655 todo was raised. Is used to create a permalink url to that line.
656 assignees (list[str]): The assignees of the issue.
657 labels (list[str]): The labels of the issue.
658 milestone (Union[None, str]): The milestone of the issue.
659 title (str): The title of the issue, once the block input
660 argument has been parsed.
661 body (Union[str, list[str]]): The body of the issue. Can be empty string
662 (no body_, or list of str for every line in body.
664 """
666 def __init__(self, line, block, hunk, file):
667 """Instantiate the ToDo Class.
669 Args:
670 line (unidiff.Line): The line from which the todo was raised.
671 block (str): The complete indented block, the ToDo was raised from.
672 Including the title.
673 hunk (unidiff.Hunk): The hunk of diff from wich the todo was triggered.
674 file (unidiff.File): The file, from which the diff was
675 extracted.
677 """
678 self.line = line
680 if line.is_added:
681 self.status = LineStatus.ADDED
682 else:
683 self.status = LineStatus.DELETED
685 self.block = block.strip()
686 self.markdown_language = 'python'
687 self.hunk = ''.join([l.value for l in hunk.target_lines()])
688 if self.hunk.count('"""') == 1:
689 if '"""' in self.hunk[:int(len(self.hunk) / 2)]:
690 self.hunk = self.hunk + '"""\n'
691 else:
692 self.hunk = '"""\n' + self.hunk
693 self.file_name = file.target_file.lstrip('b/')
694 self.target_line = line.target_line_no
696 # parse the block
697 self._parse_block()
699 def _parse_block(self):
700 """Parses the `block` argument and extacts more info."""
701 lines = self.block.split('\n')
702 lines, self.assignees = _get_assignees(lines)
703 lines, self.labels = _get_labels(lines)
704 lines, self.milestone = _get_milestone(lines)
705 if len(lines) > 1:
706 self.title, self.body = lines[0].lstrip(), '\n'.join(lines[1:])
707 self.body = [line.lstrip() for line in self.body.splitlines()]
708 else:
709 self.title = lines[0].lstrip()
710 self.body = ""
712 @property
713 def issue(self):
714 issue = Issue(
715 title=self.title,
716 labels=['todo'] + self.labels,
717 assignees=self.assignees,
718 milestone=self.milestone,
719 body=self.body,
720 hunk=self.hunk,
721 file_name=self.file_name,
722 start_line=self.target_line,
723 markdown_language=self.markdown_language,
724 status=self.status
725 )
726 return issue
728 def __bool__(self):
729 return True
732class TodoParser:
733 """Class that parses git diffs and looks for todo strings.
735 First things first, the Todos can be skipped with a syntax similar to doctest.
736 Adding 'todo: +SKIP' to the line skips the todo. The example
737 todos which follow now all contain this SKIP command, because we don't want
738 this class to raise issues in the explanatory section. Here are examples of
739 how to put todos that will be picked up by this parser:
741 Examples:
742 A single commented line starting with todo (case insensitive).::
744 # todo: This needs to be looked into. todo: +SKIP
746 Using parentheses, you can assign people to todos. If these people are
747 part of the github repo, they will be assigned to the issue, that is
748 raised from this todo. Please use their github username::
750 # todo (tensorflower-gardener): Tensorflow's bot should fix this. todo: +SKIP
752 With extended syntax you can add existing labels and exisitng milestones
753 to your todos.::
755 # todo: This is the title of the todo. todo: +SKIP
756 # Further text is indented by a single space. It will appear
757 # In the body of the issue. You can add assignees, and labels
758 # with this syntax. The `todo` label will automatically
759 # added to your labels.
760 # assignees: github_user, kevinsawade, another_user
761 # labels: devel, urgent
762 # milestones: alpha
764 Besides these in-comment todos, this class also scans for google-style
765 todo's in python docstrings.::
767 def myfunc(arg1):
768 '''This is a docstring.
770 Args:
771 arg1: A thing.
773 Todo:
774 * Single-line Todo. Add more documentation. todo: +SKIP
775 * (kevinsawade) We should also add some actual code. todo: +SKIP
776 * Multi-line todos should follow google-styleguide. todo: +SKIP
777 This means a tab should be used as indentation inside
778 the docstrings. This will form the body of the issue.
779 Assignees and labels can be added the same way:
780 assignees: github_user2, user3
781 labels: wontfix, devel
782 milestones: alpha
784 '''
786 Attributes:
787 issues (list[Issue]): A list of Issue instances.
788 testing (bool): Whether testing is carried out.
789 repo (str): A url to the current repo.
791 """
793 def __init__(self, testing=0, client=None):
794 """Instantiate the TodoParser class.
796 Keyword Args:
797 testing (bool, optional): Whether testing is carried out with this
798 class. Defaults to False.
799 client (GitHubClient): Instance of github client to precent multiple
800 instantiation.
802 """
803 self.testing = testing
804 self.issues = []
805 if client is None:
806 self.client = GitHubClient(testing=self.testing)
807 else:
808 self.client = client
809 self._parse()
811 def _parse(self):
812 """Parse the diffs and search for todos in added lines."""
813 # read env variables
814 self.repo = os.environ['INPUT_REPO']
815 self.sha = os.environ['INPUT_SHA']
816 self.before = os.environ['INPUT_BEFORE']
817 if 'INPUT_INCLUDE_TODO_AFTER_CODE_LINE' in os.environ:
818 if isinstance(os.environ['INPUT_INCLUDE_TODO_AFTER_CODE_LINE'], str):
819 self.include_todo_after_code_line = os.environ['INPUT_INCLUDE_TODO_AFTER_CODE_LINE'] in ['true', 'True']
820 else:
821 self.include_todo_after_code_line = os.environ['INPUT_INCLUDE_TODO_AFTER_CODE_LINE']
822 else:
823 self.include_todo_after_code_line = False
825 # get before and current hash
826 if self.testing == 1:
827 self.sha = '036ef2c'
828 self.before = '11858e4'
829 self.diff = self.client.get_specific_diff(self.before, self.sha)
830 elif self.testing == 2:
831 self.sha = '7fae83c'
832 self.before = '036ef2c'
833 self.diff = self.client.get_specific_diff(self.before, self.sha)
834 elif self.testing == 3:
835 self.sha = '67b1e23'
836 self.before = '63fa247'
837 self.diff = self.client.get_specific_diff(self.before, self.sha)
838 else:
839 self.diff = self.client.get_last_diff()
841 # create patchset from diff
842 patchset = PatchSet(self.diff)
844 for file in patchset:
845 # handle the file before
846 file_before = file.source_file.lstrip('a/')
847 if file_before == 'dev/null':
848 file_before = StringIO('')
849 else:
850 file_before = StringIO(self.client.get_file_at_commit(file_before, self.before))
852 # handle the file after
853 file_after = file.target_file.lstrip('b/')
854 if not file_after.endswith('.py'):
855 continue
856 file_after = StringIO(self.client.get_file_at_commit(file_after, self.sha))
858 # parse before and after todos
859 with file_before as f:
860 todos_before = extract_todos_from_file(f.read(), self.testing, self.include_todo_after_code_line)
861 with file_after as f:
862 todos_now = extract_todos_from_file(f.read(), self.testing, self.include_todo_after_code_line)
864 # iterate over hunks and lines
865 for hunk in file:
866 lines = list(hunk.source_lines()) + list(hunk.target_lines())
867 for i, line in enumerate(lines):
868 if block := is_todo_line(line, todos_before, todos_now, self.testing):
869 todo = ToDo(line, block, hunk, file)
870 issue = todo.issue
871 self.issues.append(issue)
874def is_todo_line(line, todos_before, todos_now, testing=0):
875 """Two cases: Line starts with any combination of # Todo, or line starts with
876 asterisk and is inside a napoleon docstring todo header.
878 Also filter out todo: +SKIP.
880 Args:
881 line (unidiff.Line): A line instance.
882 todos_before (list[str]): Todos from the source file.
883 todos_now (list[str]): Todos from the target file.
885 Keyword Args:
886 testing (bool, optional): Set True for Testing. Defaults to False.
888 Returns:
889 str: Either an empty string (bool('') = False), when line is not a
890 todo line, or a str, when line is a todo line. todo: +SKIP.
892 """
893 if testing == 2 and line.value == 'I will add many.':
894 print(line)
895 raise Exception("STOP")
896 # check if line has been added or removed
897 if line.is_context:
898 return ''
899 elif line.is_added:
900 todos = todos_now
901 else:
902 todos = todos_before
904 # check whether line can be a todo line
905 if re.search(INLINE_TODO_PATTERN, line.value, re.MULTILINE | re.IGNORECASE) and not TODO_SKIP_SUBSTRING in line.value:
906 stripped_line = strip_line(line.value.replace('#', '', 1))
907 elif re.search(DOCSTRING_TODO_PATTERN, line.value, re.MULTILINE | re.IGNORECASE) and not TODO_SKIP_SUBSTRING in line.value:
908 stripped_line = strip_line(line.value.replace('*', '', 1), with_todo=False)
909 else:
910 return ''
912 # get the complete block
913 # and build complete issue
914 check = [stripped_line in t for t in todos]
915 if any(check):
916 index = check.index(True)
917 block = todos[index]
918 return block
919 else:
920 return ''
923def extract_todos_from_file(file, testing=0, include_todo_after_code_line=False):
924 """Parses a file and extracts todos in google-style formatted docstrings.
926 Args:
927 file (str): Contents of file.
929 Keyword Args:
930 testing (bool, optional): If set to True todos containing `todo: +SKIP`
931 will be disregarded. Defaults to False.
933 Returns:
934 list[str]: A list containing the todos from this file.
936 """
938 # use ast to parse
939 docs = []
940 parsed_file = ast.parse(file)
941 try:
942 docs.append(ast.get_docstring(parsed_file))
943 except Exception as e:
944 raise Exception("Exclude this specific exception") from e
945 for node in parsed_file.body:
946 try:
947 docs.append(ast.get_docstring(node))
948 except TypeError:
949 pass
951 # append docstring todos to list:
952 todos = []
953 for doc in docs:
954 if doc is None:
955 continue
956 blocks = doc.split('\n\n')
957 for block in blocks:
958 if 'Todo:' in block:
959 block = '\n'.join(block.splitlines()[1:])
960 block = block.split('* ')[1:]
961 if not testing:
962 block = list(filter(lambda x: False if TODO_SKIP_SUBSTRING in x else True, block))
963 block = [strip_line(line, with_todo=False) for line in block]
964 block = list(map(lambda x: x.replace('\n ', '\n'), block))
965 todos.extend(block)
967 # get all comments lines starting with hash
968 if not include_todo_after_code_line:
969 comments_lines = list(
970 map(
971 lambda x: x.strip().replace('#', '', 1),
972 filter(
973 lambda y: True if y.strip().startswith('#') else False,
974 file.splitlines())))
975 else:
976 comments_lines = []
977 for line in file.splitlines():
978 if '# todo' in line.lower():
979 comments_lines.append(line.split('#', 1)[-1].strip().replace('#', '', 1))
981 # iterate over them.
982 for i, comment_line in enumerate(comments_lines):
983 print(comment_line, not testing and TODO_SKIP_SUBSTRING in comment_line)
984 if not testing and TODO_SKIP_SUBSTRING in comment_line:
985 continue
986 if comment_line.lower().strip().startswith('todo'):
987 block = [strip_line(comment_line)]
988 in_block = True
989 j = i + 1
990 while in_block:
991 try:
992 todo_body = comments_lines[j].startswith(' ')
993 except IndexError:
994 in_block = False
995 continue
996 if todo_body:
997 block.append(strip_line(comments_lines[j]))
998 j += 1
999 else:
1000 in_block = False
1001 block = '\n'.join(block)
1002 todos.append(block)
1004 # and return
1005 return todos
1008def strip_line(line, with_whitespace=True, with_todo=True):
1009 """Strips line from unwanted whitespace. comments chars, and 'todo'.
1011 Args:
1012 line (str): The line to be stripped.
1014 Keyword Args:
1015 with_whitespace (bool, optional): Whether to strip the whitespace
1016 that follows the comment character '#'.
1017 Defaults to True.
1018 with_todo (bool, optional): Whether to replace case insensitive
1019 'todo' strings.
1021 Returns:
1022 str: The stripped line.
1024 """
1025 if with_whitespace:
1026 line = line.strip().lstrip('#').lstrip()
1027 else:
1028 line = line.strip().lstrip('#')
1029 if with_todo:
1030 return re.split(r'(?i)todo(\s|\:)', line, 1, re.IGNORECASE | re.MULTILINE)[-1].strip()
1031 else:
1032 return line
1035################################################################################
1036# Main
1037################################################################################
1040def run_tests_from_main():
1041 """Runs unit-tests when `main()` is called with testing = True."""
1042 # load a file with secrets if there
1043 try:
1044 with open('secrets', 'r') as f:
1045 gh_token = f.readline()
1046 os.environ['INPUT_TOKEN'] = gh_token
1047 except FileNotFoundError:
1048 pass
1050 # run unittests
1051 loader = unittest.TestLoader()
1052 test_suite = loader.discover(start_dir=os.path.join(os.getcwd(), 'tests'),
1053 top_level_dir=os.getcwd())
1054 runner = unittest.TextTestRunner()
1055 result = runner.run(test_suite)
1057 return result
1060def main(testing): # pragma: no cover
1061 if testing or os.getenv('INPUT_TESTING') == 'true':
1062 if not os.path.isfile('tests/test_todo_to_issue.py'):
1063 raise Exception("Please switch the TESTING argument in your workflow.yml file to 'false'. Tests will only run in the python-todo-to-issue repository.")
1064 result = run_tests_from_main()
1065 if not result.wasSuccessful():
1066 print("Tests were not successful. Exiting.")
1067 exit(1)
1068 else:
1069 print("Tests were successful")
1070 else:
1071 if 'INPUT_INCLUDE_TODO_AFTER_CODE_LINE' in os.environ:
1072 todo_after_code_line = os.environ['INPUT_INCLUDE_TODO_AFTER_CODE_LINE'] == 'true'
1073 else:
1074 todo_after_code_line = False
1075 if todo_after_code_line:
1076 print("Checking todos that occur after code lines.")
1077 else:
1078 print("Not Checking todos that occur after code lines.")
1079 from pprint import pprint
1080 client = GitHubClient()
1081 issues = client.existing_issues
1082 todo_parser = TodoParser(client=client)
1083 print('complete diff: ', todo_parser.diff)
1084 issues = todo_parser.issues
1085 print('all issues: ', issues)
1086 for i, issue in enumerate(issues):
1087 print(f"Processing issue {issue}.")
1088 if issue.status == LineStatus.ADDED:
1089 status_code = client.create_issue(issue)
1090 if status_code is None:
1091 pass
1092 else:
1093 if status_code == 201:
1094 print('Issue created')
1095 else:
1096 print(f'Issue could not be created. The status code is {status_code}')
1097 elif issue.status == LineStatus.DELETED and os.getenv('INPUT_CLOSE_ISSUES') == 'true':
1098 status_code = client.close_issue(issue)
1099 if status_code is None:
1100 pass
1101 if isinstance(status_code, tuple):
1102 if status_code[0] == 201:
1103 print('Issue closed')
1104 else:
1105 print(f"Could not close issue. The status code is {status_code[0]}")
1106 if status_code[1] == 201:
1107 print('Added close comment to issue')
1108 else:
1109 print(f"Could not add a comment to issue. The status code is {status_code[1]}")
1110 else:
1111 if status_code == 201:
1112 print('Issue closed')
1113 else:
1114 print(f"Could not close issue. The status code is {status_code}")
1115 sleep(1)
1116 print("Finished working through the issues.")
1118if __name__ == "__main__":
1119 import argparse
1121 parser = argparse.ArgumentParser(description="Python code Todos to github issues.")
1122 parser.add_argument('--testing', dest='testing', action='store_true',
1123 help="Whether a testing run is executed and tests will be conducted.")
1124 parser.set_defaults(testing=False)
1125 args = parser.parse_args()
1126 main(testing=args.testing)