Mike Driscoll: PyDev of the Week: Valentin Haenel

This week we welcome Valentin Haenel (@esc___) as our PyDev of the Week! Valentin is a core developer of Numba and several other packages that you can see either on his website or on Github. He has also given several talks at various conferences in Europe. Let’s spend some time getting to know Valentin better!

Can you tell us a little about yourself (hobbies, education, etc):

I went to the University of Edinburgh to get a bachelor in computer science and to the Bernstein Center in Berlin to get a master in computational neuroscience. I tend to favour more traditional computer science topics these days such as compression algorithms and compilers. In my spare time, I spend time with my lovely wife Gloria, fly quad-line sports kites and ride longboards through Berlin. I’ve been doing Python and open-source on Github for about 10 years.

Why did you start using Python?

I first started using Python as part of my Masters program. Python was—and still is—quite popular in computational neuroscience, both for doing machine learning on sensor data such as EEG and fMRI and also for simulating neural models and networks of neurons. I had been using Java before and it took some getting used to the dynamic (duck) typing style. As part of the academic work I came in touch with the early scientific stack, which at the time consisted mostly of Numpy, Scipy, Matplotlib and the command-line IPython shell. Some of my earliest Python work from that time still survives. A project I did to simulate spiking neurons using a specific type of model:

https://github.com/esc/molif — this was my first github repo ever.

Also from that time is the first of my packages to make it into Debian, a Python interface to a specific type of hardware photometer. In fact, I just checked on this Ubuntu machine (Mar 2019), the package is still available:

$   apt search pyoptical   Sorting... Done   Full Text Search... Done   python-pyoptical/bionic,bionic 0.4-1.1 all     python interface to the CRS 'OptiCAL' photometer     :)

What other programming languages do you know and which is your favorite?

I know a little C, shell, go and Java, but Python is by far my favorite though. A friend of mine is working on a secret programming language project called ‘@’, which aims to be… well… runtime only — very intriguing.

What projects are you working on now?

I am now working on Numba for Anaconda Inc. Besides that, I am also working on Blosc (http://blosc.org/) including python-blosc and Bloscpack. In addition, there are a few smaller, but somewhat popular, projects that I run by myself, namely wiki2beamer, git-big-picture, conda-zsh-complation and yadoma. Most recently, I have been getting more interested in time tracking and stared using and contributing to Færeld.

Which Python libraries are your favorite (core or 3rd party)?

I have always had an interest in crafting command line interfaces. I have looked into many libraries for this task such as getopt, optparse, argparse, bup/options.py, miniparser, opster, blargs, plac, begins and click (have I forgot any?!). However, the one library that I keep coming back to and the one that I recommend over all others is docopt: http://docopt.org/ . There is something to be said about designing your command line interface as a program synopsis and then getting a fully fledged parser from just that. For me personally, this is the fastest and most natural, intuitive and convenient way to construct a command line argument parser. If you are not aware of it yet, you should definitely go and check it out!

How you get involved with Numba?

I saw an opening for a software engineering position at Anaconda Inc. doing mostly open source work on Numba. Working low-level and on a compiler is right down my alley and was something I had been wanting to do for a very long time. I applied, they made me an offer, the rest is history.

Can you explain why you would use Numba versus PyPy or Cython?

Cython is a superset of Python, it has additional syntax that allows for static typing which will then compile the code to run at C-speed, a.k.a to “cythonize” the code. That code can no longer run as regular Python code. Numba is much less invasive than this but has a similar goal. It provides the `@jit` decorator which allows Numba to perform Just In Time (JIT) type inference and compilation using the LLVM compiler infrastructure under the hood. Importantly, it does this on the Python bytecode, does not require any types to be annotated and the code can continue to run as regular Python (once you comment out the `@jit` decorator.) This has the advantage, that you can ship portable numeric code as pure Python with only Numba as a dependency which will significantly reduce your packaging and distribution overhead. Both Cython and Numba have been used in the scientific space traditionally. This is because they interact well with the existing ecosystem, the native libraries (where Cython can even interface with C++ which Numba can not) and are designed to be strongly aware of Numpy. So those are the ones you would use when working in that space: for example machine learning and broadly speaking any scientific algorithms and simulations. PyPy on the other hand has traditionally not had good support for the whole scientific stack. It is a bit better nowadays (early 2019) as both Numpy and Pandas can be compiled and a lot of work has gone into
making c-extensions work in PyPy.

Anyway, the primary goal of PyPy focuses on moving beyond CPython (the C implementation of the Python interpreter) as a base for a Python programs and it is slowly but surely getting there.

So, in conclusion: PyPy is the future of the Python language in general but it is not quite ready for data-intensive applications. If you want to have as much computational efficiency as possible today, then both Numba and Cython are good choices. Numba is very easy to try out—just decorate your bottlenecks—and has been known to accelerate code by one or two orders of magnitude.

What advice do you have for new people who want to start helping an open source project?

Go find yourself an itch; find a project in whatever your favorite language that you find personally useful and improve it. Then, contribute your changes back. Chances are, if it is useful for you, it will be useful for other people. Also, because it is useful to you personally, you are likely to continue contributing to it because you end up having a vested interest in it. And so obviously, personal utilities are a great category to go looking for such tools. Find something that is useful for you on a day-to-day basis and contribute to that. Also, don’t be afraid to put your code out there in the open and don’t let yourself be discouraged if your contributions are rejected, you are just at the beginning of your journey, so keep going. Good luck!

Is there anything else you’d like to say?

A big thank you goes out to all the open source/free software developers and contributors out there. I am very proud to be a part of this fantastic and inspirational community.

Thanks for doing the interview, Valentin!

The post PyDev of the Week: Valentin Haenel appeared first on The Mouse Vs. The Python.

Planet Python

Mike Driscoll: Book Review: Practical Python and OpenCV

I bought Practical Python and OpenCV a couple of years ago during one of its authors Kickstarters. I started reading it and then got busy with other things. The past couple of weeks, I decided to give the book another go and was able to finish it. Note that I started reading the 3rd edition of the book without realizing there was a 4th edition. After finished the 3rd edition, I compared it to the 4th side by side and it looks like they are nearly identical, so I don’t think it matters all that much.

Quick Review

  • Why I picked it up: Computer vision / machine learning sounds interesting to me and the author has a fun blog
  • Why I finished it: It’s short and the writing style is engaging
  • I’d give it to: Anyone looking to get started with OpenCV in Python

  • Book Formats

    You can get this book as a hard cover, PDF, or Mobi.

    Book Contents

    This book contains 11 chapters across 169 pages. It covers Python 3 and OpenCV 4 in the 4th edition.

    Full Review

    Practical Python and OpenCV is an interesting book. It’s shorter than your normal technical book, but it covers a lot of ground. This is kind of an anomaly as a lot of technical books seem to be much longer. Anyway, let’s go over the book piece by piece.

    Chapter one is just an introduction. It describes what computer vision is an gives some examples. Then chapter two jumps in and teaches you how to install the packages you need to use the book effectively. Personally I think these two chapters could have been combined or the installation chapter could have been an appendix. But no matter.

    Chapter three is where you will finally get into some code. Here you learn how to load images into OpenCV and display them to the user. You will also learn how to save the image. This is useful for converting between different image formats, but more importantly it is helpful for saving off your data when you need to.

    In the fourth chapter, you learn the basics of images from the point of view of OpenCV. You will learn its coordinate system, as well as how to access and manipulate individual pixels in an image.

    Then in chapter five, you discover how to draw lines, rectangles and circles. You can change the width, position, and color of each of these shapes. You can also stack the shapes on each other.

    Chapter six covers the topic of image processing. Here you will learn all about image transformations (translation, rotation, resizing, etc), image arithmetic, bitwise operations, masking, splitting, merging and the importance of color spaces (RGB vs BGR and HSV).

    For chapter seven, you will learn how to use histograms effectively in OpenCV. The histogram will help you determine the contrast, brightness and intensity distributions of the images that you are working with. OpenCV supports working with grayscale and color histograms. You will also learn about histogram equalization and masks.

    Chapter eight explains the different types of blurring and smoothing that OpenCV can do. Here you will learn about Gaussian, Median and Bilateral blurring. Adrian doesn’t go into a lot of depth explaining these topics. He instead opts to show the reader how to do each of them using small code snippets.

    Chapter nine covers the topic of thresholding, which is the binarization of images. Thresholding is the term used to describe focusing on objects or areas of interest within an image. Adrian uses a photo of some coins for this chapter and the rest of the book. You end up using thresholding to find the coins in the photos.

    Then in chapter ten, you learn how to use gradients and edge detection to refine what you learned in chapter nine. This will allow you to find the edges of the coins in the photo. This chapter focuses on using the Laplacian and Sobel methods for gradients. Then you learn about using the Canny Edge detection technique.

    To wrap up the book, Adrian uses OpenCV to find the contours of the coins so that he can count the coins in the image. He also talks about the differences that exist between OpenCV versions when it comes to finding contours.

    This book covers a lot of different topics very quickly. There are some minor layout issues in my mind in that sometimes he refers to an image that either doesn’t appear soon or was inexplicably shown earlier in the chapter. There are also a couple of occasions where he is explaining some code that is not on the page. However these things don’t affect the quality of the code examples themselves. You also get a Case Studies PDF that has more information.

    Overall, I think this is a pretty good book. The examples are fun and fast paced and seeing dinosaurs in a Python book is always fun. The book is kind of pricey though.

    Practical Python and OpenCV cover

    Practical Python and OpenCV

    by Dr. Adrian Rosebrock


    Other Book Reviews

    The post Book Review: Practical Python and OpenCV appeared first on The Mouse Vs. The Python.

    Planet Python

    Mike Driscoll: PyDev of the Week: Sean McManus

    This week we welcome Sean McManus (@musicandwords) as our PyDev of the Week! Sean is the author of several books, including Mission Python: Code a Space Adventure Game!, which was reviewed on this site in March. There are free chapters from his book available here. You can learn more about Sean on his website. Let’s take some time to get to know him better!

    Can you tell us a little about yourself (hobbies, education, etc):

    I’m a writer specialising in technology. In recent years I’ve written several books to get children and young adults into coding. The launch of the Raspberry Pi and initiatives such as Code Club have helped to make coding much more accessible to young people than it was for many years.

    As a kid, I loved programming my Amstrad CPC computer and in many ways it started me on my career path. I had listings and articles published in the leading magazines of the day, and my first book was about Amstrad programming. Today, I can still remember how much I loved programming as a kid, and I hope that my books bring some of that excitement to today’s budding coders.

    As well as writing books and articles, I help some of the world’s leading technology companies with their copywriting requirements.

    Outside of writing, I enjoy photography and making music. I have recorded an album of electronic music I plan to release online later this year, and I am a member of a singing group which is a great way to unwind.

    How did you end up writing a book about games programming in Python?

    It seems like there are lots of Python books that include small examples to show you how a particular feature works. That’s great, because it’s much easier to learn when the code is pared down to the essentials. However, that does leave many readers wondering what to do next, and how you bridge the gap between short programs of a couple of pages and more substantial projects. It’s one thing to know how a list works, for example, but how do you use that to create a map for a 3D game? How do you build on the basics to create something that does more than just demonstrate the language?

    With Mission Python, the idea was to show a worked example of a game that goes beyond the basics. It’s a graphical adventure game, so there are lots of opportunities to learn about data structures (rooms, objects, interactions) and lots of customisation opportunities that you wouldn’t get with a simple arcade game. The game is set on a Martian space station where the air is leaking and you have to get to safety. It involves finding objects and solving puzzles, and uses a forced perspective like some of the early Zelda games. I was particularly pleased with a comment from one reviewer saying that it felt like the game came before the book, rather than the other way around as is often the case with educational books: I was keen for the game to be as “real” as possible, within the constraints of the slightly retro game format and what can be reasonably documented within the scope of a book.

    Which Python libraries are your favorite (core or 3rd party)?

    The game in Mission Python uses the Pygame Zero library, so I’ve been spending a lot of time working with that and consider it my current favourite library.

    Pygame Zero modernises Pygame with some education-friendly features, and simplified syntax for getting some images moving. It helps to make the journey from Scratch to Python smoother in the classroom by introducing Actors, which are like sprites. It also simplifies some of the basics of Pygame, such as screen refreshes. There are lots of cool things Pygame Zero can do that were outside the scope of Mission Python, so perhaps I’ll code something else with it in the future.

    What lessons did you learn in writing a book?

    I’ve written or co-written several books now, including Raspberry Pi For Dummies, Scratch Programming in Easy Steps and Cool Scratch Projects in Easy Steps. Writing books teaches you to be organised with your time and with your ideas: it’s an intense period of work in a short space of time, often covering a wide range of topics, so planning is essential. I track everything in spreadsheets, to take the weight off my brain!

    One of my tips is to set aside thinking time as much as possible. If you’re really stumped when debugging, I’ve found it’s often better to switch to doing something else, to buy some thinking time in the background without stalling the project totally. People often have “aha!” moments in the shower or when they are otherwise relaxed, so if tinkering with the code isn’t working, it’s a good idea to let it rest for a bit.

    Thinking time is important for creativity too: I set aside an evening to design the main game puzzles. I went to a late session in the space gallery at the Science Museum in London, where I spent the evening drawing up puzzles and locations, surrounded by real rockets and astronaut suits. I later added new puzzles and ideas while developing the game, but it helped to have a good vision from the outset.

    The books have obviously taught me a lot about the programming languages I’ve been working with, as I’ve looked into new features and resources that I need. Writing a book, I have to take a pragmatic view and sometimes I prefer to use a solution that is easier for someone to understand or one that is more widely compatible over one that might be the fastest or most idiomatic.

    What has the response been to Mission Python?

    Mission Python was a big project, writing both the game and the accompanying book, so it’s wonderful that it is out there now in the world. I’ve been getting some lovely feedback. A young reader wrote to tell me they hadn’t programmed before and this book introduced them to Python programming, which is a wonderful thing to have played a part in. Adults have been enjoying the book too, some of whom knew Python but hadn’t done any game development before, as well as some who are new to Python entirely. Hobbyist coders who love programming are finding it’s an enjoyable project, and it’s been wonderful to see tweets and reviews from people who are reading the book. It’s a huge help with getting the word out and is greatly appreciated. The game engine can be used to create new games entirely, so I’m looking forward to seeing what readers build with it.

    What projects are you working on now?

    My main project right now is updating my website, which involves a lot of code wrangling, albeit not Python. My website celebrated its 20th birthday last year, so there’s some quite old code in there that I’m weeding out as I work to make the site more responsive and mobile-friendly.

    I’m still tinkering with the game from Mission Python, both for enjoyment and to provide additional resources for readers. For example, NASA released some real sounds recorded on Mars so that seemed like too good an opportunity to miss. I’ve written some code so that those sounds play when you’re on the surface of Mars in the game. I have a short routine I’ll be publishing soon to hide objects on the space station, once my new website design launches, which brings a new puzzle element to the game.

    I’ll be spending more time with Scratch again this year too, as I update my Scratch books for Scratch 3.0.

    I’m looking forward to getting my teeth into another big Python project as soon as I have the time to dedicate to it. I consider myself on a journey with Python, and am keen to further develop my skills.

    Is there anything else you’d like to say?

    Anyone is welcome to download the Mission Python game here. There’s a free sample of the book there too, which includes the instructions for playing the game, and there’s a map too for if you get stuck!

    Thank you for inviting me to feature on your blog, Mike!

    Thanks for doing the interview, Sean!

    The post PyDev of the Week: Sean McManus appeared first on The Mouse Vs. The Python.

    Planet Python

    Mike Driscoll: How to Extract Build Info from Jenkins with Python

    I work with continuous integration software as a part of my job. I use both Hudson and Jenkins in my role and occasionally need to interact with them programmatically. There are two Python packages you can use for this task:

    The Python Jenkins package will work with both Hudson and Jenkins which JenkinsAPI only works with Jenkins. I usually use Python Jenkins because of this, although I have recently started looking to see which one works better with artifacts and I discovered that JenkinsAPI is actually better for that sort of thing. So you will need to evaluate both of these packages depending on what you need to do.

    Install Python Jenkins

    To follow along with the code examples in this article, you will need to install Python Jenkins. You can use pip for that:

    pip install python-jenkins

    Now that it’s installed, let’s give Python Jenkins a whirl!

    Getting All the Jobs from Jenkins

    One common task is needing to get a listing of all the jobs that are configured in your build system.

    To get started, you need to login to your Jenkins server:

    import jenkins   server = jenkins.Jenkins('http://server:port/', username='user',                           password='secret')

    Now you have a Jenkins object that you can use to execute REST requests against your Jenkins CI server. The results that are returned are usually a Python dictionary or dictionary of dictionaries.

    Here’s an example getting all the jobs that are configured on your CI system:

    import jenkins   server = jenkins.Jenkins('http://server:port/', username='user',                          password='secret')   # Get all builds jobs = server.get_all_jobs(folder_depth=None) for job in jobs:     print(job['fullname'])

    This will loop through all the jobs that are configured in Jenkins and print out their job names.

    Getting Job Information

    Now that you know the names of the jobs on your Jenkins box, you can get more detailed information about each job.

    Here’s how:

    import jenkins   server = jenkins.Jenkins('http://server:port/', username='user',                          password='secret')   # Get information on specific build job # This returns all the builds that are currently shown in  # hudson for this job info = server.get_job_info('job-name')   # Passed print(info['lastCompletedBuild'])   # Unstable print(info['lastUnstableBuild'])   # Failed print(info['lastFailedBuild'])

    The get_job_info() will give you a lot of information about the job, including all the currently saved builds. It is nice to be able to extract which builds have passed, failed or are unstable.

    Getting Build Information

    If you want to know how long a job takes to run, then you need to get down to the build level.

    Let’s find out how:

    import jenkins   server = jenkins.Jenkins('http://server:port/', username='user',                          password='secret')   info = server.get_job_info('job-name')   # Loop over builds builds = info['builds'] for build in builds:     for build in builds:         print(server.get_build_info('job-name',                                      build['number']))

    To get build metadata, you need to call get_build_info(). This method takes in the job name and the build number and returns the metadata as a dictionary.

    Wrapping Up

    You can do a lot more with the Python Jenkins package. For example, you can use it to start a build job, create a new job or delete an old one as well as quite a few other things. Unfortunately, the documentation is pretty bare-bones, so you’ll have to do a fair bit of experimentation to get it working the way you want.

    Additional Reading

    Planet Python

    Mike Driscoll: PyDev of the Week: Tania Allard

    This week we welcome Tania Allard (@ixek) as our PyDev of the Week! Tania is a developer advocate at Microsoft. She is also a speaker at multiple conferences. If you’d like to learn more about her, you should check out her blog. She also has some of her projects up on Github for you to peruse. Let’s take a few moments to get to know Tania!

    Can you tell us a little about yourself (hobbies, education, etc):

    I am originally from Mexico but have lived in the USA and in the UK for the last 8 years.

    I have a bachelor’s in Mechatronic engineering and have also always been fascinated by technology and I can class myself as a lifelong learner. As such I got a PhD from the University of Manchester in Data science applied to Materials science, during which I discovered and fell in love with Python. Since completing my PhD I have worked as a research software engineer, research engineer, data engineer, and more recently could advocate.

    Apart from tech I love Olympic weightlifting, so I spend quite a good amount of time in the gym every week and I am already looking forward for this year’s competition season!

    I also love craft beer and recently joined the women in beer scene in Manchester, UK where I live.

    Why did you start using Python?

    As I said before I did a PhD in Materials Science, but instead of focusing on the experimental side of things my work was focused on modelling materials for tissue replacement. Most of the people in my discipline were using MATLAB for such purposes (as did I at the beginning) but eventually I realized I needed something more flexible. I was also soon driven into the ‘open science’ movement and decided I should create not only all my analyses but also my thesis plots and research papers outputs using an open source programming language to encourage reproducibility and accessibility.
    It was until I starts using Python that I discovered the open source community and never looked back.

    What other programming languages do you know and which is your favorite?

    Coming from a scientific computing background I know MATLAB, FORTRAN (spoiler alert I do think modern FORTRAN is really good), C/C++, R, Julia, and Assembly. More recently I have been diving into functional programming via Scala and also into Go.

    What projects are you working on now?

    I recently joined the developer advocate team at Microsoft, so I am focusing on understanding the community barriers and needs when it comes to cloud computing.
    In the Python ecosystem I am trying to work more closely with the Jupyter community to help drive better accessibility and community initiatives.

    Something I am really excited about is the organisation of the mentored sprints for underrepresented folks in the Python community during PyCon.

    Finally, I am working on a project focused on understanding the relationship between corruption and gender inequality in Latin American countries such as Mexico and Colombia. I truly believe that we can make a positive societal impact through technology, so this project is something I really care about.

    What non-Python open source projects do you enjoy using?

    As an R user I really like the Drake project as it has provided important tools towards reproducible scientific workflows (which is also a thing I deeply care about). I also have been enjoying working with projects like Docker, Kubernetes and Terraform over the last couple of years.

    How did you get into speaking at Python events?

    Since I come from a research background, I was already familiar with talks, poster presentations, conferences and the such, but I never thought I could speak at Python events. Then I joined a local scientific Python group and was eventually asked to speak about my PhD work. I was terrified back then, but after giving my talk and realising that people actually cared about the topic and were super supportive I decided I would like to share my knowledge with others.

    Do you have any advice for others who would like to start giving talks?

    If you are wondering whether you have something interesting to say I can guarantee you do!

    Local user groups/meetups are great places to get started at within a relaxed environment. Folks are usually very supportive and eager to see what is going on in the community.

    Is there anything else you’d like to say?

    Thanks for the interview Mike. Hope to see loads of folks around in PyCon and if you’re there come say hi!

    Thanks for doing the interview, Tania!

    Planet Python