Refactor: feedback on metrics email structure by MarceloRobert · Pull Request #1791 · kernelci/dashboard

MarceloRobert · 2026-03-11T20:35:35Z

Changes

Changed build issues query to return "number of unique issues in the interval" + "number of unique issues that had their first incident in the interval" + "total number of incidents in the interval"
Added subquery to return "which issue had the most incidents in the interval grouped by origin" (limited to top 3) and added that section to the email
Changed lab query to make the build count be "builds related to tests performed by each lab" instead of "builds directly related to each lab", and removed origin grouping
Updated email formatting
Updated unit tests accordingly

How to test

This PR changes the existing queries and format of the metrics summary email, so to test it just connect to a database and run poetry run python3 manage.py notifications --action metrics_summary

Closes #1780

Example with real data

KernelCI Metrics Summary
========================
Period: 2026-03-26 19:46 UTC to 2026-04-02 19:46 UTC


  COVERAGE
  --------
                      This week    Last week    Change
                      ─────────    ─────────    ──────
  Trees monitored     50           44           +6
  Checkouts           4,030        4,004        +26  (+1%)
  Builds              11,777       7,730        +4,047  (+52%)
  Tests               784,875      422,611      +362,264  (+86%)


  BUILD REGRESSIONS
  -----------------
  A "regression" is defined as a reported problem affecting 0 or multiple builds.

  Origin    Regressions       Affected           Affected builds by top issues
            (known + new)     Builds (total)     #1      #2      #3
  ─────────────────────────────────────────────────────────────────────────
  maestro   16  + 27  = 43    438                63      45      37      
  redhat    1   + 1   = 2     9                  6       3       
  ─────────────────────────────────────────────────────────────────────────
  Total     17  + 28  = 45    447


  TOP REGRESSIONS PER ORIGIN
  --------------------------
  maestro:
    #1 (63 occurrences) -  ‘done’ is used uninitialized [-Werror=uninitialized] in drivers/cp...
      Dashboard: https://d.kernelci.org/issue/maestro:e84b665ddd4424ed72a8bbede90636bdce186f99?iv=1
    #2 (45 occurrences) -  'HMAC_Update' is deprecated [-Wdeprecated-declarations] in crypto/...
      Dashboard: https://d.kernelci.org/issue/maestro:cfb7db7bad026f784ab5093df91515e27c634b34?iv=1
    #3 (37 occurrences) -  alignment may not be specified for ‘pidfd’ in drivers/block/zram/z...
      Dashboard: https://d.kernelci.org/issue/maestro:5a652717b9a5090a029aaeee303245bd07ee330c?iv=1
  
  redhat:
    #1 (6 occurrences) -  error: target emulation unknown: -m or at least one .o file requir...
      Dashboard: https://d.kernelci.org/issue/redhat:a8b090056d8e6bc4eee9ffa4389946c992a6e989?iv=1
    #2 (3 occurrences) -  ‘at91_default_usrio’ undeclared here (not in a function) in driver...
      Dashboard: https://d.kernelci.org/issue/redhat:e385e5050284ef5d130dbb06ffd633dadfd81c18?iv=1
  

  TEST LABS ACTIVITY
  ------------------
  8 labs reported results this week (1 more than last week).

  Labs marked with an asterisk (*) are new.

  Lab                   Covered builds  Boots    Tests          Change (tests)
  ────────────────────────────────────────────────────────────────────────────────
  lava-broonie          522             2,392    279,594        +108,272  (+63%)
  lava-cip              643             1,562    251            +179  (+249%)
  lava-collabora        913             5,517    318,892        +169,708  (+114%)
  lava-foundriesio      170             233      0              0  (0%)
  lava-kci-qualcomm *   48              122      75,087         +75,087
  lava-pengutronix      360             950      0              0  (0%)
  opentest-ti           46              1,477    35             +35
  pull-labs-demo        586             913      2,401          -182  (-7%)
  ────────────────────────────────────────────────────────────────────────────────
  Total                 3,288           13,166   676,260        +353,099  (+109%)

--
This is an experimental report format. Please send feedback in!
Talk to us at kernelci@lists.linux.dev

Made with love by the KernelCI team - https://kernelci.org

==============================================

MarceloRobert · 2026-03-17T19:17:02Z

Paused work on this PR in order to fix cronjob issues and deployment issues

tales-aparecida · 2026-03-19T18:13:18Z

Some feedback from TSC today: https://docs.google.com/document/d/1z5EVstx7N57FKxQ1loY5FtVKjz1xvi-G0Wwq5qDoX8Q/edit?tab=t.0#bookmark=kix.y0sdm2heccwl

maestro 16 (+16) is that number real? Shouldn't the first number be higher (counting all issues in the database)?

TOP REGRESSIONS PER ORIGIN: @broonie suggested adding a human readable description for each issue. Try to show the title/name/summary of each issue alongside their URL. Let's see how crowded it gets. Add a reasonable char limit.

TEST LABS ACTIVITY: Some labs apparently have more builds than boots. Is that accurate?

tales-aparecida · 2026-03-19T18:15:44Z

@broonie you mentioned some confusion regarding the scope of

  lava-broonie             1246            1,116    297,163        -71,584  (-19%)

I don't quite got what you were wandering in regard to maestro builds. Do you have a specific question that the team could answer?

broonie · 2026-03-19T18:21:35Z

Other people were confused about why there are more builds than boots, people were wondering if it was showing only maestro generated builds of if this is also including builds from my non-maestro infrastructure as well.

MarceloRobert · 2026-03-26T11:24:21Z

Other people were confused about why there are more builds than boots, people were wondering if it was showing only maestro generated builds of if this is also including builds from my non-maestro infrastructure as well.
- @broonie

The previous numbers were just temporary data for the example, they weren't extracted from a real query. The current example now reflects real data, and there are fewer builds than boots

MarceloRobert · 2026-03-26T11:26:00Z

TOP REGRESSIONS PER ORIGIN: broonie suggested adding a human readable description for each issue. Try to show the title/name/summary of each issue alongside their URL. Let's see how crowded it gets. Add a reasonable char limit.
- @tales-aparecida

Done, added the issue comment to the list, see if looks ok

MarceloRobert · 2026-03-26T11:27:45Z

maestro 16 (+16) is that number real? Shouldn't the first number be higher (counting all issues in the database)?
- @tales-aparecida

I'll check it, it should be counting all issues from that origin within the interval and then counting how many of those were the first incident of an issue. Should we count the full number of issues on that origin?

tales-aparecida · 2026-03-26T12:10:16Z

maestro 16 (+16) is that number real? Shouldn't the first number be higher (counting all issues in the database)?

@tales-aparecida

I'll check it, it should be counting all issues from that origin within the interval and then counting how many of those were the first incident of an issue. Should we count the full number of issues on that origin?

I think the word "Total" could be misleading in the table header. Maybe "Reported", or something that suggests that we hit that many regressions during the filtered interval

alanpeixinho

LGTM

alanpeixinho · 2026-03-27T20:11:31Z

backend/kernelCI_app/management/commands/notifications.py


+    # Compute the text spacing for the labs so it isn't too big or too small
+    lab_spacing = max((len(lab_key) for lab_key in data.lab_maps.keys()), default=0)
+    lab_spacing += 5  # add spacing for leading tab and possible * mark for new labs


just a nit, but a little more spacing would improve readability

MarceloRobert · 2026-04-01T11:34:39Z

@tales-aparecida @bhcopeland do you guys have any comments?

bhcopeland · 2026-04-01T11:43:26Z

@tales-aparecida @bhcopeland do you guys have any comments?

Can we update the issue with a new version of the report with the changes?

MarceloRobert · 2026-04-01T12:10:39Z

Can we update the issue with a new version of the report with the changes?

@bhcopeland the description of the pull request already has an updated example, do you mean adding this example to the issue that the PR closes?

bhcopeland · 2026-04-01T12:21:54Z

Can we update the issue with a new version of the report with the changes?

@bhcopeland the description of the pull request already has an updated example, do you mean adding this example to the issue that the PR closes?

Ah, okay, apologies, I misread it, then yes LGTM. As for the other comment, I leave that up to you. The report is looking really good. I'll bring it up in TSC tomorrow, but from my side, we are in the final stretch. Great work on this. Thank you.

One quick question for you. On maestro I see you have 17 (+7), under Build Regerssion. Which means that 7 have been added during that period (I checked upstream and can see these reports). What is the 17 counting? That bit isn't clear to me. Can we add a bit to the report to make it clear what those two numbers represent?

MarceloRobert · 2026-04-01T13:37:34Z

One quick question for you. On maestro I see you have 17 (+7), under Build Regerssion. Which means that 7 have been added during that period (I checked upstream and can see these reports). What is the 17 counting? That bit isn't clear to me. Can we add a bit to the report to make it clear what those two numbers represent?

Those numbers are the total unique issues that had incidents during that period (17), and number of unique issues that had their first incident during that period (+7). Would you say that fits as a good caption?

bhcopeland · 2026-04-01T14:26:22Z

One quick question for you. On maestro I see you have 17 (+7), under Build Regerssion. Which means that 7 have been added during that period (I checked upstream and can see these reports). What is the 17 counting? That bit isn't clear to me. Can we add a bit to the report to make it clear what those two numbers represent?

Those numbers are the total unique issues that had incidents during that period (17), and number of unique issues that had their first incident during that period (+7). Would you say that fits as a good caption?

Okay, so total regressions found with existing known issues were 17, and new known issues created were 7. So, in total, 24 regressions have been found, of which 7 are not-seen-before. Maybe a simple (Existing, New) under Reported Regressions would clear that up?

tales-aparecida · 2026-04-01T15:33:06Z

Okay, so total regressions found with existing known issues were 17, and new known issues created were 7. So, in total, 24 regressions have been found, of which 7 are not-seen-before. Maybe a simple (Existing, New) under Reported Regressions would clear that up?

The meaning is: We hit 17 different regressions, out of which 7 are new

bhcopeland · 2026-04-01T15:54:46Z

Okay, so total regressions found with existing known issues were 17, and new known issues created were 7. So, in total, 24 regressions have been found, of which 7 are not-seen-before. Maybe a simple (Existing, New) under Reported Regressions would clear that up?

The meaning is: We hit 17 different regressions, out of which 7 are new

Yeah, I feel we should add (Existing|New) to the report. Next to Reported Regressions. Apologies, my maths was a bit off, as I added 17 and 7.

Reported Regressions (Existing, New)

tales-aparecida · 2026-04-01T16:01:41Z

How about a mathy format:

  BUILD REGRESSIONS
  -----------------
  A "regression" is defined as a reported problem affecting 0 or multiple builds.

  Origin    Regressions         Affected           Affected builds by top issue
         Known + New = Total   Builds (total)     #1      #2      #3
  ─────────────────────────────────────────────────────────────────────────
  maestro   10 + 7 = 17          171                31      30      16      
  redhat     1 + 0 =  1            6                 6       
  ─────────────────────────────────────────────────────────────────────────
  Total     11 + 7 = 18          177

MarceloRobert · 2026-04-01T16:02:55Z

Yeah, I feel we should add (Existing|New) to the report. Next to Reported Regressions. Apologies, my maths was a bit off, as I added 17 and 7.

Another alternative would be 17 (7 new) for having less change

The thing is that these 7 new ones are part of the total of 17; so IIUC the existing | new in this case would be 10 | 7, which is not bad tbh

MarceloRobert · 2026-04-02T13:47:49Z

@tales-aparecida @bhcopeland I'll add this Existing/New separation and merge it later since this email is sent every Saturday and we'll have a holiday tomorrow. If there's more feedback on the TSC meeting we can adress them in a separate PR, sounds good?

MarceloRobert · 2026-04-02T19:50:06Z

Updated the PR and the example in the PR description

Changes build query to return n_issues, n_new_issues and total_incidents; Added subquery to return top 3 issues for each origin; Changed lab query to return builds related to the tests of each lab (instead of builds directly related to the lab). Closes kernelci#1780

MarceloRobert self-assigned this Mar 11, 2026

MarceloRobert added Queries Issue that involves modifying some DB query Metrics Related to open metrics, measurements or usage data labels Mar 11, 2026

MarceloRobert force-pushed the refactor/metrics-email-feedback branch 2 times, most recently from 40fd35b to 0fe1489 Compare March 25, 2026 20:34

MarceloRobert force-pushed the refactor/metrics-email-feedback branch from 0fe1489 to 908a36a Compare March 26, 2026 21:04

MarceloRobert marked this pull request as ready for review March 26, 2026 21:04

tales-aparecida requested review from bhcopeland and tales-aparecida March 27, 2026 13:06

alanpeixinho approved these changes Mar 27, 2026

View reviewed changes

tales-aparecida approved these changes Apr 1, 2026

View reviewed changes

MarceloRobert force-pushed the refactor/metrics-email-feedback branch from 908a36a to 397a656 Compare April 2, 2026 19:48

MarceloRobert force-pushed the refactor/metrics-email-feedback branch from 397a656 to 225b9b0 Compare April 2, 2026 19:57

Conversation

MarceloRobert commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

How to test

Example with real data

Uh oh!

MarceloRobert commented Mar 17, 2026

Uh oh!

tales-aparecida commented Mar 19, 2026

Uh oh!

tales-aparecida commented Mar 19, 2026

Uh oh!

broonie commented Mar 19, 2026

Uh oh!

MarceloRobert commented Mar 26, 2026

Uh oh!

MarceloRobert commented Mar 26, 2026

Uh oh!

MarceloRobert commented Mar 26, 2026

Uh oh!

tales-aparecida commented Mar 26, 2026

Uh oh!

alanpeixinho left a comment

Choose a reason for hiding this comment

Uh oh!

alanpeixinho Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

MarceloRobert commented Apr 1, 2026

Uh oh!

bhcopeland commented Apr 1, 2026

Uh oh!

MarceloRobert commented Apr 1, 2026

Uh oh!

bhcopeland commented Apr 1, 2026

Uh oh!

MarceloRobert commented Apr 1, 2026

Uh oh!

bhcopeland commented Apr 1, 2026

Uh oh!

tales-aparecida commented Apr 1, 2026

Uh oh!

bhcopeland commented Apr 1, 2026

Uh oh!

tales-aparecida commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarceloRobert commented Apr 1, 2026

Uh oh!

MarceloRobert commented Apr 2, 2026

Uh oh!

MarceloRobert commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MarceloRobert commented Mar 11, 2026 •

edited

Loading

tales-aparecida commented Apr 1, 2026 •

edited

Loading