Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Take a look at 28DAU metrics drop #5760

Closed
tejal29 opened this issue Apr 29, 2021 · 5 comments
Closed

Take a look at 28DAU metrics drop #5760

tejal29 opened this issue Apr 29, 2021 · 5 comments
Assignees
Milestone

Comments

@tejal29
Copy link
Member

@tejal29 tejal29 commented Apr 29, 2021

We are seeing 28DAU metrics drop internally from 62K to 56K

Please take a look at it.

Thanks
Tejal

@MarlonGamez
Copy link
Contributor

@MarlonGamez MarlonGamez commented May 4, 2021

Took a look last thursday and again today.
Last thursday I did the following:

  • checked last 5 binaries to see if there were issues with the update check functionality (the latest release did not have issues)
  • checked previous runs of 28DAU scripts for errors (didn't see any)
  • checked for a similar drop in open telemetry metrics (didn't see any)

As of today there are some updates:

  • the 1DAU seems to be recovering, but we'll have to see how it continues. The metrics outage from that view seems to be a random anomaly
  • We also are seeing issues with open telemetry metrics, possible related to our keys. We are investigating this currently.
@MarlonGamez
Copy link
Contributor

@MarlonGamez MarlonGamez commented May 6, 2021

It seems like the keys for all versions before v1.23.0 are expiring, and v1.23.0 generated a new key to use. Because of this, we're seeing metrics from all previous versions come to a halt.

@tejal29
Copy link
Member Author

@tejal29 tejal29 commented May 6, 2021

Thanks @marlon-gamez for all the updates!!

@aaron-prindle
Copy link
Contributor

@aaron-prindle aaron-prindle commented May 11, 2021

Update - New strategy here is to fix potential timing issues of service account keys by using API keys. See #5812 for the tracking bug w/ more information

@aaron-prindle aaron-prindle modified the milestones: v1.24.0, v1.25.0 May 11, 2021
@tejal29
Copy link
Member Author

@tejal29 tejal29 commented May 11, 2021

Closing this now. The 28DAU is back up to 61K.

@tejal29 tejal29 closed this May 11, 2021
@aaron-prindle aaron-prindle removed their assignment May 11, 2021
@tejal29 tejal29 modified the milestones: v1.25.0, v1.24.0 May 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants