Artem Rys: Monitoring traffic of your Github repositories using Python and Google Cloud Platform — Part 1

Monitoring traffic of your Github repositories using Python and Google Cloud Platform — Part 1

Photo by Paweł Czerwiński on Unsplash

It is an article about monitoring your Github open-source repositories traffic. Unfortunately, you can see these statistics only by accessing each repository step by step. You may not want to access them at all… But if you do, you can use this small tool.

Technical stack:

And from the perspective of $  — this solution is zero cost because of the free quota you have in Google Cloud Platform. No ads — I just like using free opportunities.

So, the main concept is to get top referrers from Github for each your public repository and then store this data in firestore based on the date to be able to create a report on the weekly basis (in the next part).

In this part, we are going only to get top referrers from Github. The main and only code:

Cloud Function to get top referrers for each of your open-source Github repositories.

requirements.txt file:

PyGithub==1.43.8
google-cloud-firestore==1.4.0

You are going to need a Github personal access token to be able to make requests to Github API to get all your public repositories and then get traffic data from them. You can get it here.

Github personal access token page.

Click Generate new token.

Github new personal access token page. Click only `public_repo` scope.

Click Generate token.

From the GCP perspective — you need to have a created project, enabled Cloud Functions API and created a Cloud Firestore database and that’s all.

Docs to deploy Cloud Function in several ways can be found here.

I am using a gcloud tool from my local machine.

gcloud config set core/project <your-project-name>
gcloud functions deploy parse_github_repos_traffic --runtime python37 --trigger-http --set-env-vars=GITHUB_TOKEN=<your-github-token>

Newly created Cloud function.

To test the function, click on it, then go to the Testing tab.

A testing tab of your Cloud Function.

And press Test the function. And you should be able to see something like this.

A testing tab of your Cloud Function after pressing a Test the function button.

And this data is also available in Cloud Firestore (for future analytics).

Cloud Firestore with data.

That is all for this part. In the next part we are going to set up a scheduler to run call this function weekly and analyze the data we got to create a report.

Thanks for the attention to the topic, feel free to leave your questions in the comments for discussion.


Monitoring traffic of your Github repositories using Python and Google Cloud Platform — Part 1 was originally published in python4you on Medium, where people are continuing the conversation by highlighting and responding to this story.

Planet Python

PSF GSoC students blogs: Google Summer of Code with Nuitka 5th Weekly Check-in

1. What did you do this week?
This week, I continued to work on my script which automates the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440

The script automates the manual testing of comparing pytest results of a nuitka compiled wheel using `python setup.py bdist_nuitka` to the pytest results of an uncompiled wheel built using `python setup.py bdist_wheel` for the most popular PyPI packages. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

Testing has been improved and extended to many more PyPI packages.

2. What is coming up next?
Perfect the script in preparation for merging and work on documentations.

3. Did you get stuck anywhere?
Some PyPI packages are special and require special handling during automation. I skipped these for the sake of speed, but I will need to get back to them in the future.

Planet Python

PSF GSoC students blogs: Google Summer of Code with Nuitka 4th Blog Post

<meta charset=”utf-8″>

<meta charset=”utf-8″>

This week, I continued to work on my script which automates the testing of nuitka-wheel pytest. Details can be found on my pull request: https://github.com/Nuitka/Nuitka/pull/440

 

The script automates the manual testing of comparing pytest results of a nuitka compiled wheel using `python setup.py bdist_nuitka` to the pytest results of an uncompiled wheel built using `python setup.py bdist_wheel` for the most popular PyPI packages. Testing is done to ensure that nuitka is building the wheel correctly. If the pytests pass/fail in the same way, that means Nuitka built the wheel properly. Else if the tests differ, then something is wrong. Virtualenv is used to create a clean environment with no outside pollution.

 

There were a lot of changes made this week.

  • Added an `__import__` check to make sure pytest used the correct (uncompiled or compiled) package

  • Added a mechanism to ignore certain tests which are unimportant (fixed the comparison issue for urllib3)

  • Added coloring to output for better visuals

  • Added a summary of all packages at the very end of testing

  • Extended testing to many more packages

  • Added local caching of `git clone`

  • Many run time improvements

 

Very proud of the work I have done this week. The plan for next week is to extend the automation to more PyPI packages.

 

Planet Python