63 Commits

Author SHA1 Message Date
daveoconnor
00c5be94d4 Release import adjustments (#2045) 2025-12-16 16:13:03 -05:00
daveoconnor
5fa57839da Celery workflow changes for release reports (#2043) 2025-12-12 13:12:01 -08:00
daveoconnor
c980879935 Release Report Enhancements (#1750) (#1992) 2025-11-04 14:14:20 -08:00
daveoconnor
da39d8e8c2 Add "Import new releases" emails (#1872) (#1945) 2025-10-20 09:53:48 -07:00
Dave O'Connor
c6335c2fe2 Updated import commands to allow and default to 'new' functionality. Remove artifactory-related commands. (#1744) 2025-08-27 11:19:03 -04:00
Greg Kaleka
e3a057120a Docs fixes and fix bug in library import (#1788) 2025-05-19 15:13:22 -04:00
Greg Kaleka
5610f14f4c Fix word cloud font issue and improve logs (#1747) 2025-04-12 19:36:40 -04:00
Greg Kaleka
6b58dc3692 Improve ml fetching efficiency during release tasks (#1746) 2025-04-12 17:31:48 -04:00
daveoconnor
da6ee3524e Addtitional Mailing List stats in release report (#1712) 2025-03-26 18:08:20 -07:00
Greg Kaleka
8b43040399 Website traffic stats gathering (#1679) 2025-03-23 16:11:21 -04:00
Greg Kaleka
d537985bb8 Schedule report tasks and add report preview view (#1620) 2025-02-10 11:22:17 -05:00
daveoconnor
c4e6302360 Financial steering committee page member images (#1613) 2025-01-27 10:18:23 -08:00
Brian Perrett
a08c01617a Add management command to run all necessary tasks after a release. (#1562) 2024-12-20 13:01:33 -08:00
Brian Perrett
2981976b80 Add functionality to track LibraryVersion dependencies (#1524) 2024-12-16 14:28:33 -05:00
Brian Perrett
646f916ee6 Updates and additions to release report stats (#1401)
- Add basic line change counts for each x.x.0 library versions.
- import line change counts during `import_commits` task.
- Add `update_issues` management command for importing Issues for each
LibraryVersion.
- Add update issues runnable command in `Issue` admin.
- Build word cloud using mailinglist content.
- Update release report with new data.
2024-10-31 10:14:41 -07:00
Brian Perrett
e230128f61 Add authors to LibraryVersions. (#1347)
- fixes #1325
- reuses the logic that imports authors for libraries to do the same for
LibraryVersions.
- Authors are only added if an email is listed for the author in the
metadata
2024-10-10 10:39:16 -07:00
Brian Perrett
48c09d3d5e Import commits per release and create release reports (#1263)
View stats per release, we do this by
doing log diffs between release tags. Ex: `git log
boost-1.78.0..boost-1.79.0`. The output is parsed and the commits
are saved with a foreign key to the `LibraryVersion` it relates to.

- commits are imported by doing "bare" clones (no project files, only
git data) of repos into temporary directories, as created by python's
bulitin `tempfile.TemporaryDirectory`
- Added Commit model
- Added CommitAuthor model
- Added CommitAuthorEmail model
  - One CommitAuthor can have many emails.
- Added task for importing commits. (and admin link to trigger it)
- Added task for importing CommitAuthor github data (avatar and profile
url, with admin link to trigger it)
- Added a basic Library stat page which can be viewed by going to the
admin -> library -> view stats.
- Added a `Get Release Report` button in the `LibraryAdmin` which allows
a staff member to select a boost version and up to 8 libraries to
generate a report for. The report is just a webpage which attempts to
convert cleanly to a pdf using the browser's print to pdf functionality.
- Updated the Library Detail page to show commits per release instead of
per month.
- Updated the Library Detail page to show `Maintainers & Contributors`
sorted by maintainers, then the top contributors for the selected
release, then the top contributors overall by commits descending.
- Removed CommitData, which was tracking monthly commit stats
2024-09-25 15:09:07 -07:00
Lacey Williams Henschel
e5268ed7d8 Fix Container Hash and other docs:
- Add LibraryVersion inline to LibraryAdmin for easier debugging
- Add convenience function to generate random string
- Adjust Library slug so it functions as unique without being unique
- Add exception logic for older Container Hash versions
- Fix bugs in management commands
- Add exception logic for older Container Hash docs
- Add exception logic for Functional/Overloaded Function docs
- Add task to import all library-versions, and add that to existing admin refresh buttons
- Add logic to import github_url if we dont have it
2024-02-15 10:08:03 -08:00
Lacey Williams Henschel
468615dad8 Fix bug where commit counts for libraries were not being updated monthly as they should have been
- Remove the kwargs from the call to update_libraries
- Add placeholder task for commit counts
- Refactor to simplify the commit count logic and pull it out of the management command and into a task
- Adjust schedule
- Add a separate task to update current month commit counts
- Add CommitData to admin, and add functionality to update the CommitData from the admin via button, like other admin features
2024-02-12 11:32:09 -08:00
Lacey Williams Henschel
863bca5005 - Add API endpoint to import versions
- Move import library versions logic into task
- Add API docs
2023-09-28 09:40:12 -07:00
Lacey Williams Henschel
214f37986b Streamline import commands and add docs
Add data JSONField to LibraryVersion model

Add fields to library list admin display

Add commands to update authors and maintainers individually

Exclude data JSONField from view querysets

Silence some output from the library-version import management command

Remove unused field from the library-version import management command

Save library-versions more cleanly in the library-version import management command

Remove loading maintainers from the import command, since they now have their own command

Add docs for new commands

Add boost_setup command to run one command to import all data

Add docs on first-time data import

Better exception handling, quieter flow, reduce GH API calls

Graceful handling if there is not a github repo

Pass most recent 12 months to commit counts command

Add some user-friendly output to setup command
2023-09-19 11:17:05 -07:00
Lacey Williams Henschel
87347c7f05 Remove first github release date; unused 2023-09-12 10:36:12 -07:00
Lacey Williams Henschel
4712551827 Improvements to import/update libraries command
- Remove outdated options, quiet some logging
- Don't add authors yet -- need to add maintainers first
- Add GitHub token option to update_libraries
2023-09-07 15:58:07 -07:00
Lacey Williams Henschel
5646fa9ea4 Correct setting name 2023-09-07 14:24:58 -07:00
Lacey Williams Henschel
f57b5dcade Add setting for minimum Boost version to import 2023-09-07 14:09:40 -07:00
Lacey Williams Henschel
7e847cc17f - Move GitHub retrieval and parser classes to the core/ app from the libraries/ app; moved tests, too, and updated import statements
- Add `verbose` flag to `import_versions` to allow user to silence the output
- Removed some unused options from `import_versions`
- Moved the exclusion logic in `import_versions` to its own function
- Stop getting the `github_url` from the github API. Instead, generate it from the tag. (reason: in older tags, the best URL GitHub gives us is a less-useful one to a _commit_ and not a tag. We can generate the url, though.)
- Move the retrieval of the `release_date` to its own task, so it can be loaded async. (reason: it's a separate API call per version to get the date)
- Make `release_date` optional on the `Version` model (reason: make the field easier to load async)
- Simplify logic to retrieve `release_date` and just always retrieve it from the commit
- Stop loading the version `description` from GitHub. We don't use it on the frontend anyway, and in all but the most recent couple of versions, the `description` is the comment from the commit, which is usually not useful.
2023-08-31 15:32:16 -07:00
Lacey Williams Henschel
44db4ac03d Import maintainers when there are no maintainers 2023-08-30 10:40:18 -07:00
Lacey Williams Henschel
fcec414638 Correct the default minimum for the import script 2023-08-21 12:20:12 -07:00
Lacey Williams Henschel
eeb615ceff Load docs urls for library-versions from S3
- Add documentation_url to LibraryVersion table
- Retrieve the documentation_url in the view
- If the documentation_url is not present, show the Version documentation_url
- Hide the "https://" in the template
- Add Celery task to retrieve the documentation_urls from the libraries page in S3 and match them to import_library_versions
- Add HTML helper function to extract the libraries and docs URLs from the HTML
- Add one-time command to import docs urls
- Load the documentation_urls in the command that loads all library-versions
- Add a min-release option to the command that loads library-versions
- Document the new command and changes to the existing command
2023-08-04 08:40:00 -07:00
Lacey Williams Henschel
6a2a2e4ceb Remove sample data commands 2023-08-03 16:17:06 -07:00
Lacey Williams Henschel
da1a28451d Skip time-consuming github data for local dev 2023-07-21 10:32:37 -07:00
Lacey Williams Henschel
9f5db36d3f Try to get the library-version repo with another field if it fails with the submodule name 2023-07-20 09:23:00 -07:00
Lacey Williams Henschel
dd039d20c2 Catch the error with version 1.61.0 2023-07-20 09:23:00 -07:00
Lacey Williams Henschel
f8475f1009 Refactor to fetch API commit data in method that adds the CommitData objects 2023-07-10 06:22:36 -07:00
Lacey Williams Henschel
b26c319bb0 Fix bug related to getting tag info in get_ref 2023-07-10 06:22:36 -07:00
Lacey Williams Henschel
f319378d45 Refactor to add libraries if they don't exist 2023-07-10 06:22:36 -07:00
Natalia
a8cd8f507a Ran pre-commit run -a
Completed runs of black and rest of pre-commit checks.
2023-05-30 23:21:53 -03:00
Lacey Williams Henschel
e238649e58 Add management command to import commit counts 2023-05-26 09:48:55 -07:00
Lacey Williams Henschel
7911928f4f Add management command to import release dates 2023-05-19 12:55:02 -07:00
Lacey Williams Henschel
49850d4261 Code simplifications 2023-05-18 09:25:20 -07:00
Lacey Williams Henschel
2499c4891e Style changes, changes per PR review 2023-05-18 09:25:20 -07:00
Lacey Williams Henschel
dedd103807 Add command to import library-versions 2023-05-18 09:25:20 -07:00
Lacey Williams Henschel
488ca4d3ba Final command, docs, linter 2023-05-15 10:19:32 -07:00
Lacey Williams Henschel
fc63727f0d 🔥 Should not be in this PR 2023-05-15 10:19:32 -07:00
Lacey Williams Henschel
b0cf084090 WIP load library release dates 2023-05-15 10:19:32 -07:00
Lacey Williams Henschel
eb0a6884b6 168 import versions (#295)
* WIP get old versions

* 🏦 Add URL and data dict to Version model

* 🔧 Some refactoring how we get tag data

* First pass at uploading versions

* 🔧 Move parsing logic to the parser and add tests

* Retrieve url and save json data

* Improve version upload docs

* Linter
2023-05-11 05:59:49 -07:00
Lacey Williams Henschel
57753b0d74 🔥 Remove dead code 2023-03-27 12:39:34 -07:00
Lacey Williams Henschel
385518620f 📝 Address libraries_json lists, fix extract_names to accommodate short strings 2023-03-21 10:56:54 -07:00
Lacey Williams Henschel
f64c2ad405 📝 Changes find_user method to find_contributor 2023-03-21 10:42:55 -07:00
Lacey Williams Henschel
f719893264 Merge branch 'main' of github.com:CPPAlliance/boost.org into 144-upload-contributors 2023-03-20 11:14:30 -07:00