Catalin George Festila: Python 3.7.3 : Using the pycryptodome python module.

This python module can be used with python 3. More information can be found here. PyCryptodome is a self-contained Python package of low-level cryptographic primitives. It supports Python 2.6 and 2.7, Python 3.4 and newer, and PyPy. The install of this python module is easy with pip tool: C:\Python373\Scripts>pip install pycryptodome Collecting pycryptodome … Installing collected packages:
Planet Python

Weekly Python StackOverflow Report: (clxxxii) stackoverflow python report

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2019-06-15 20:04:06 GMT


  1. How to use random to choose colors – [11/3]
  2. Conditional Cumulative Sums in Pandas – [9/3]
  3. Efficient way to loop over 2D array – [9/2]
  4. Will passing ignore_index=True to pd.concat preserve index succession within dataframes that I’m concatenating? – [9/1]
  5. How do I properly use a function under a class? – [8/4]
  6. Why are f-strings faster than str() to parse values? – [7/1]
  7. How to efficiently calculate triad census in undirected graph in python – [6/4]
  8. Convert elements of list in pandas series using a dict – [6/3]
  9. Python Regex escape operator \ in substitutions & raw strings – [6/2]
  10. Rewrite to dictionary comprehensions – [6/2]

Planet Python

Python Anywhere: Using MongoDB on PythonAnywhere with MongoDB Atlas

.jab-post img { border: 2px solid #eeeeee; padding: 5px; }

This requires a paid PythonAnywhere account

Lots of people want to use MongoDB with PythonAnywhere; we don’t have support for it built in to the system, but it’s actually pretty easy to use with a database provided by MongoDB Atlas — and as Atlas is a cloud service provided by Mongo’s creators, it’s probably a good option anyway 🙂

If you’re experienced with MongoDB and Atlas, then our help page has all of the details you need for connecting to them from our systems.

But if you’d just like to dip your toe in the water and find out what all of this MongoDB stuff is about, this blog post explains step-by-step how to get started so that you can try it out.

Prerequisites

The first important thing to mention is that you’ll need a paid PythonAnywhere account to access Atlas. Free accounts can only access the external Internet using HTTP or HTTPS, and unfortunately MongoDB uses it’s own protocol which is quite different to those.

Apart from that, in order to follow along you’ll need at least a basic understanding of PythonAnywhere, writing website code and of databases in general — if you’ve been through our Beginners’ Flask and MySQL tutorial you should be absolutely fine.

Signing up to MongoDB atlas

Unsurprisingly, the first step in using Atlas is to sign up for an account (if you don’t already have one). Go to their site and click the “Try Free” button at the top right. That will take you to a page with the signup form; fill in the appropriate details, and sign up.

Atlas’ user interface may change a little after we publish this post (their sign-up welcome page has already changed between out first draft yesterday and the publication today!), so we won’t give super-detailed screenshots showing what to click, but as of this writing, they present you with a “Welcome” window that has some “help getting started” buttons. You can choose those if you prefer, but we’ll assume that you’ve clicked the “I don’t need help getting started” button. That should land you on a page that looks like this, which is where you create a new MongoDB cluster.

Creating your first cluster and adding a user

A MongoDB cluster is essentially the same as a database server for something more traditional like MySQL or PostgreSQL. It’s called a cluster because it can potentially spread over multiple computers, so it can in theory scale up much more easily than an SQL database.

To create the cluster:

  • Choose the “AWS” cloud provider, then select the region that is closest to where your PythonAnywhere account is hosted:
    • If your account is on our global site at www.pythonanywhere.com, choose the us-east-1 region.
    • If your account is on our EU site at eu.pythonanywhere.com, choose the eu-central-1 region.
  • Click the “Create cluster” button at the bottom of the page.

This will take you to a page describing your cluster. Initially it will have text saying something like “Your cluster is being created” — wait until that has disappeared, and you’ll have a page that will look something like this:

Now we need to add a user account that we’ll use to connect to the cluster from Python:

  • Click the “Database Access” link in the “Security” section of the menu on the left-hand side, which will take you to a page where you can administer users.
  • Click the “Add new user” button on the top right of the pane that appears.
  • Enter a username and a password for the new user (make sure you keep a note of these somewhere)
  • The “User Privileges” should be “Read and write to any database”
  • The “Save as a temporary user” checkbox should not be checked.

Click the button to create the user, and you’ll come back to the user admin page, with your user listed in the list.

Setting up the whitelist

Access to your MongoDB cluster is limited to computers on a whitelist; this provides an extra level of security beyond the username/password combination we just specified.

Just to get started, we’ll create a whitelist that comprises every IP address on the Internet — that is, we won’t have any restrictions at all. While this is not ideal in the long run, it makes taking the first steps much easier. You can tighten things up later — more information about that at the end of this post.

Here’s how to configure the whitelist to allow global access:

  • Click on the “Network Access” link in the “Security” section of the menu on the left-hand side, which will take you to the page where you manage stuff like whitelists.
  • Click the “Add IP Address” button near the top right of the page.
  • On the window that pops up, click the “Allow access from anywhere” button. This will put “0.0.0.0/0” in the “Whitelist entry” field — this is the CIDR notation for “all addresses on the Internet”.
  • Put something like “Everywhere” in the “Comment” field just as a reminder as to what the whitelist entry means, and leave the “Save as temporary whitelist” checkbox unchecked.
  • Click the “Confirm” button.

Getting the connection string

Now we have a MongoDB cluster running, and it’s time to connect to it from PythonAnywhere. The first step is to get the connection string:

  • Click on the “Clusters” link in the “Atlas” section of the menu on the left-hand side, which will take you back to the page you got when you first created the cluster:

  • Click the “Connect” button just underneath the name of the cluster (probably “Cluster0”).
  • You’ll get a window with different connection options; click the “Connect your application” option.
  • In “Choose your driver version”, select a “Driver” of “Python” and a version of “3.6 or later”.
  • Once you’ve done that, in the “Connection String Only” section, you’ll see something like mongodb+srv://admin:<password>@cluster0-zakwe.mongodb.net/test?retryWrites=true&w=majority

  • Copy that string (there’s a button to do that for you) and paste it somewhere safe for later use. You’ll note that it has <password> in it; you should replace that (including the angle brackets) with the actual password that you configured for your user earlier on.
  • Now you can close the popup with the button in the bottom right.

Connecting to the cluster from a PythonAnywhere console

Next, go to PythonAnywhere, and log in if necessary. The first thing we need to do here is make sure that we have the correct packages installed to connect to MongoDB — we’ll be using Python 3.7 in this example, so we need to use the pip3.7 command. Start a Bash console, and run:

pip3.7 install --user --upgrade pymongo dnspython 

Once that’s completed, let’s connect to the cluster from a command line. Run ipython3.7 in your console, and when you have a prompt, import pymongo and connect to the cluster:

import pymongo client = pymongo.MongoClient("<the atlas connection string>") 

…replacing <the atlas connection string> with the actual connection string we got from the Atlas site earlier, with <password> replaced with the password you used when setting up the user.

That’s created a connection object, but hasn’t actually connected to the cluster yet — pymongo only does that on an as-needed basis.

A good way to connect and at least make sure that what we’ve done so far has worked is to ask the cluster for a list of the databases that it currently has:

client.list_database_names() 

If all is well, you’ll get a result like this:

['admin', 'local'] 

…just a couple of default databases created by MongoDB itself for its own internal use.

Now let’s add some data. MongoDB is much more free-form than a relational database like MySQL or PostgreSQL. There’s no such thing as a table, or a row — instead, a database is just a bunch of things called “collections”, each of which is comprised of a set of “documents” — and the documents are just objects, linking keys to values.

That’s all a bit abstract; I find a useful way to imagine it is that a MongoDB database is like a directory on a disk; it contains a number of subdirectories (collections), and each of those contains a number of files (each one being a document). The files just store JSON data — basically, Python dictionary objects.

Alternatively, you can see it by comparison with an SQL database:

  • A MongoDB database is the same kind of thing as an SQL database
  • A MongoDB collection is a bit like a table — it’s meant to hold a set of similar kinds of things — but it’s not so restrictive, and defines no columns.
  • A MongoDB document is kind of like a row in such a table, but it’s not constrained to any specific set of columns — each document could in theory be very different to all of the others. It’s just best practice for all of the documents in a collection to be broadly similar.

For this tutorial, we’re going to create one super-simple database, called “blog”. Unsurprisingly for a blog, it will contain a number of “post” documents, each of which will contain a title, body, a slug (for the per-article URL) and potentially more information.

The first question is, how do we create a database? Neatly, we don’t need to explicitly do anything — if we refer to a database that doesn’t exist, MongoDB will create it for us — and likewise, if we refer to a collection inside the database that doesn’t exist, it will create that for us too. So in order to add the first row to the “posts” collection in the “blog” database we just do this:

db = client.blog db.posts.insert_one({"title": "My first post", "body": "This is the body of my first blog post", "slug": "first-post"}) 

No need for CREATE DATABASE or CREATE TABLE statements — it’s all implicit! IPython will print out the string representation of the MongoDB result object that was returned by the insert_one method; something like this:

<pymongo.results.InsertOneResult at 0x7f9ae871ea88> 

Let’s add a few more posts:

db.posts.insert_one({"title": "Another post", "body": "Let's try another post", "slug": "another-post", "extra-data": "something"}) db.posts.insert_one({"title": "Blog Post III", "body": "The blog post is back in another summer blockbuster", "slug": "yet-another-post", "author": "John Smith"}) 

Now we can inspect the posts that we have:

for post in client.blog.posts.find():     print(post) 

You’ll get something like this:

{'_id': ObjectId('5d0395dcbf76b2ab4ed67948'), 'title': 'My first post', 'body': 'This is the body of my first blog post', 'slug': 'first-post'} {'_id': ObjectId('5d039611bf76b2ab4ed67949'), 'title': 'Another post', 'body': "Let's try another post", 'slug': 'another-post', 'extra-data': 'something'} {'_id': ObjectId('5d039619bf76b2ab4ed6794a'), 'title': 'Blog Post III', 'body': 'The blog post is back in another summer blockbuster', 'slug': 'yet-another-post', 'author': 'John Smith'} 

The find function is a bit like a SELECT statement in SQL; with no parameters, it’s like a SELECT with no WHERE clause, and it just returns a cursor that allows us to iterate over every document in the collection. Let’s try it with a more restrictive query, and just print out one of the defined values in the object we inserted:

for post in client.blog.posts.find({"title": {"$  eq": "Another post"}}):     print(post["body"]) 

You’ll get something like this:

Let's try another post 

MongoDB’s query language is very rich — you can see a list of the query operators here. We won’t go into any more detail here — there are many excellent MongoDB tutorials on the Internet, so if you google for “mongodb python tutorial” you’re bound to find something useful!

Now we’ve connected to a MongoDB database, and created some data, so let’s do something with that.

Connecting to the cluster from a Flask website

The next step is to connect to our cluster from a website’s code. We’ll use Flask for this — because it’s database-agnostic, it’s a better fit for MongoDB than Django, which is quite tied to the SQL model of representing data.

We’re also going to use a Flask extension called Flask-PyMongo to make our connections — the raw PyMongo package has a few problems with the way PythonAnywhere runs websites, and while there are ways around those (see the help page), the Flask extension handles everything smoothly for us. So, in your Bash console, exit IPython and run

pip3.7 install --user Flask-PyMongo 

Once that’s done, let’s create a website: head over to the “Web” page inside PythonAnywhere, and create yourself a Python 3.7 Flask app.

When it’s been created, edit the flask_app.py file that was auto-generated for you, and replace the contents with this:

from flask import Flask, render_template from flask_pymongo import PyMongo  app = Flask(__name__) app.config["MONGO_URI"] = "<the atlas connection string>"  mongo = PyMongo(app)  @app.route('/') def index():     return render_template("blog.html", posts=mongo.db.posts.find())  @app.route('/<post_slug>') def item(post_slug):     return render_template("blog.html", posts=mongo.db.posts.find({"slug": post_slug})) 

…replacing <the atlas connection string> with the connection string as before, with one change — the original connection string will have /test in it, like this:

mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/test?retryWrites=true&w=majority 

That /test means “connect to the database called test on the cluster”. We’ve put our data into a database called blog, so just replace the test with blog, so that it looks like this:

mongodb+srv://admin:iW8qWskQGJcEpZdu9ZUt@cluster0-zakwe.mongodb.net/blog?retryWrites=true&w=majority 

All of the rest of the code in that file should be pretty obvious if you’re familiar with Flask — the MongoDB-specific stuff is very similar to the code we ran in a console earlier, and also to the way we would connect to MySQL via SQLAlchemy. The only really new thing is the abbreviated syntax for searching for an exact match:

mongo.db.posts.find({"slug": post_slug}) 

…is just a shorter way of saying this:

mongo.db.posts.find({"slug": {"$  eq": post_slug}}) 

To go with this Flask app, we need a template file called blog.html in a new templates subdirectory of the directory containing flask_app.py — here’s something basic that will work:

<html>     <head>         <meta charset="utf-8">         <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha512-dTfge/zgoMYpP7QbHy4gWMEGsbsdZeCXz7irItjcC3sPUFtf0kuFbDz/ixG7ArTxmDjLXDmezHubeNikyKGVyQ==" crossorigin="anonymous">         <title>My blog</title>     </head>      <body>         <div class="container">             <div class="row">                 <h1><a href="/">My Blog</a></h1>             </div>              {% for post in posts %}                 <div class="row">                     <h2><a href="/{{ post.slug }}">{{ post.title }}</a></h2>                      <p>                     {{ post.body }}                     </p>                 </div>             {% endfor %}          </div>      </body> </html> 

Note that there’s one simple difference to the way we reference the post document to how we’d do it in Python code — in Python we have to say (for example) post["title"], while dictionary lookups in a Flask template require us to use post.title.

Once you’ve created that file, reload the website using the button on the “Web” page, and you should see a website with your blog posts on it:

Let’s add a new post: keep the tab showing your website open, but in another tab go to your Bash console, start ipython3.7 again, connect to the database, and add a new post:

import pymongo client = pymongo.MongoClient("<the atlas connection string>") db = client.blog db.posts.insert_one({"title": "Blog Post Goes Forth", "body": "...but I thought we were coding Python?", "slug": "bad-blackadder-nerd-joke"}) 

Head back to the tab showing the site, and hit the browser’s refresh button — your new post will appear!

All done!

So now we have a working super-simple blog running on PythonAnywhere, backed by an Atlas MongoDB cluster.

The one remaining issue is that the whitelist we specified is a little broad. If someone gets hold of your MongoDB username and password, they can access the database. It’s possible to set things up so that you have an initially-empty whitelist, and then every time you run your code, it automatically whitelists the IP address it’s running on using the Atlas API — that’s a slightly more advanced topic, though, so if you want to learn about that, head over to our MongoDB help page.

We hope this post has been useful — if you have any questions or comments, please leave them below. Also, if there are other things you’d like to connect to from PythonAnywhere that you think could benefit from having a blog post explaining how to do it, please do let us know!

Planet Python

Reuven Lerner: Playing with Python strings, lists, and variable names — or, a complex answer to a simple question

I recently received a question from a reader of my “Better developers” list. He asks:

Is there any way to turn a str type into a list type? For example, I have a list of elements, and want to turn that element into a separate list. For example, if I have

test = ['a', 'b', 'c']

I want the output to be

a=[], b=[], c=[]

One of the mantras of Python is that there should be one, and only one, way to do something. Reality has a way of being more complex than that, though, and in this particular case, the problem that my reader described in words and what he put in code weren’t exactly the same thing. (Which is a common problem in the professional software world — the specifications say one thing, but the client’s intentions say another.)

Let’s start with what my reader says he wants to do, and then get to what he actually seems to want:

He says that he wants to turn a string into a list. Well, there are a few ways to do that. The easiest is to use the “list” class, and apply it to a function:

He says that he wants to turn a string into a list. Well, there are a few ways to do that. The easiest is to use the “list” class, and apply it to a function:

>>> s = 'abc' >>> mylist = list(s) >>> mylist ['a', 'b', 'c']         

In such a case, the “list” class (which can be called, like a function, and is thus known as a “callable” in the Python world) iterates over the elements of our string. Each element is turned into a separate element in a new list that it returns.

This is fine if you want to create a new list with the same number of elements as there are characters in the string. After all, both strings and lists are Python sequences; when you create a list in this way, based on a string, you’ll find that the new list’s length and elements are identical. So s[0] and mylist[0] will return the same result, as will “len(s)” and “len(mylist)” even though “s” and “mylist” are different types.

Another way to create a list from a string is via the “str.split” method. I use this method all the time, especially when taking input from a user and iterating over the words, or fields, that the user provides. For example:

>>> words = 'here are some words' >>> words.split(' ') ['here', 'are', 'some', 'words']

The result of “str.split” is always a list of strings. And as you can see in the above example, we can tell “str.split” what string should be used as a field delimiter; “str.split” removes all occurrences of that string, returning a list of strings.

What happens if our string is a bit weird, though, such as:

>>> words = 'here    are some     words'

Now we’re going to get an equally weird result:

>>> words.split(' ') ['here', '', '', '', 'are', 'some', '', '', '', '', 'words']

This happens because “str.split” has taken our instructions very literally, as computers do: Whenever you encounter a space character, create a new element in the output list. However, this is rarely the solution that you want, and thus “str.split” has a great default: If you don’t pass anything (or pass “None” explicitly), then any length of whitespace characters will be treated as a single delimiter. Which means that we can say:

>>> words = 'here    are some     words' >>> words.split() ['here', 'are', 'some', 'words']

This is quite useful… and yet, while this is how I interpreted the question I got, it’s not what the user wants.

Rather, what he seems to want is to create new variables based on the elements of the string. So if the string is “abc”, then we want to create new variables “a”, “b”, and “c”, each of which references an empty list.

This is certainly possible, but I’ll admit it’s a bit odd. However, it gives us a chance to delve into some of Python’s more rarely used capabilities. (At least, I almost never use them — maybe other people are different!)

My first reaction to creating variables dynamically is to say, “No, you don’t really want to do that,” and to suggest that we create a dictionary, instead. You can think of a dict as your own private namespace, one which can’t and won’t interfere with the variables created elsewhere.

We could create an empty dictionary, and then iterate over the string, adding new key-value pairs to it, with each value being an empty list:

>>> for one_letter in 'abc':         d[one_letter] = []  >>> d {'a': [], 'b': [], 'c': []}

There is, however, a better way to do what we did here, and that is by using the “dict.fromkeys” class method. This is a great shortcut to creating a dictionary whose keys are known but whose values aren’t, at least not at the start. So we can say:

>>> dict.fromkeys('abc') {'a': None, 'b': None, 'c': None}

As you can see, the value associated with each key here is “None”. We don’t want that; instead, we want to have an empty list. So we can pass an empty list as a second, optional argument to “dict.fromkeys”:

>>> dict.fromkeys('abc', []) {'a': [], 'b': [], 'c': []} 

However, you should be a bit nervous before working with the dictionary I’ve created here, because every single one of the values now refers to the same list! For example:

>>> d = dict.fromkeys('abc', []) >>> d {'a': [], 'b': [], 'c': []} >>> d['a'].append(1) >>> d['b'].append(2) >>> d['c'].append(3) >>> d {'a': [1, 2, 3], 'b': [1, 2, 3], 'c': [1, 2, 3]}                

In many ways, this is similar to the problem of mutable defaults, in that we have a single value referenced in multiple places. It’s pretty obvious to experienced Python developers that this will happen, but it’s far from obvious to newcomers.

Another way to do this would be to use a dict comprehension:

>>> {one_letter : []      for one_letter in 'abc'} {'a': [], 'b': [], 'c': []}

“Wait,” you might be saying, “Maybe we have to worry about these lists also all referring to the same thing?”

Nope:

>>> d = {one_letter : []          for one_letter in 'abc'} >>> d['a'].append(1) >>> d['b'].append(2) >>> d['c'].append(3) >>> d {'a': [1], 'b': [2], 'c': [3]}         

What’s the difference between this, and our previous use of “dict.fromkeys”? The difference is that here, the “[]” empty list is evaluated anew with each iteration over the string. Thus, we get a new empty list each time. By contrast, passing the same empty list as a second argument to “dict.fromkeys” gave us the same list each time.

So if you want to use a dict — and that’s my recommendation — then you are good to go! But if you really and truly want to create variables based on the values in the string, then we’ll have to use a few more tricks.

One is to take advantage of the fact that global variables are actually stored in a dictionary. Yes, that’s right — you might think that when you write “x=100” that you’re storing things in some magical location. But actually, Python turns your variable name into a string, and uses that string as a key into a dictionary.

We don’t have direct access to this dictionary, but we can retrieve it using the “globals” builtin function. Here’s what happens when I invoke “globals” in a brand-new Python 3 interactive shell:

>>> globals() {'__name__': 'main', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>}         

See what happens now, after I assign some variables:

>>> x = 100 >>> y = [10, 20, 30] >>> z = {'a':1, 'b':2} >>> globals()          {'__name__': 'main', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'x': 100, 'y': [10, 20, 30], 'z': {'a': 1, 'b': 2}}         

Take a look at the end, and you’ll see our three newly assigned variables.

It turns out that we can also define (or update the values of) global variables in this way, too:

>>> globals()['x'] = 234 >>> globals()['y'] = [9,8,7,6] >>> globals()['z'] = 'hello out there'          >>> globals() {'__name__': 'main', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'x': 234, 'y': [9, 8, 7, 6], 'z': 'hello out there'}

I don’t really recommend this in actual code, but if you’re absolutely, positively sure that you want to do this, then you can accomplish this task in the following way:

>>> for one_letter in 'abc':     globals()[one_letter] = []         

Sure enough:

>>> x [] >>> y [] >>> z []

Again, you almost certainly don’t want to have this sort of code in production. But it does work, as we see here.

Something else we could do is use the “exec” function, which lets us run any string as a tiny Python program. We could thus say:

>> for one_letter in 'abc':         exec(f'{one_letter} = []')  >>> a [] >>> b [] >>> c []         

As you can see, it worked: We used an f-string to create a tiny (one-statement) Python program, and then used “exec” to run it. Note that we wouldn’t be able to use the related “eval” function here, because “eval” expects to have an expression, and assignment in Python isn’t an expression.

Finally, I’d generally argue that it’s a good idea not to create or manipulate global variables whose names are created dynamically from the user’s input. It’s probably best (as I wrote above) to use a dictionary. However, if you really insist on doing this, then you should probably do it in a module.

But wait — aren’t modules normally defined in files? Yes, but you can create a module on the fly by running the “module” class, just as we did above with the “list” class. There’s just one hitch, namely that the “module” class isn’t available to us in any of the Python namespaces.

That’s OK: We can grab the class via another module (e.g., __builtins__), and then invoke it, passing it the name of the module we want to create. Then we can use the builtin “setattr” function to assign a new attribute to the module. Here’s how that would look:

>>> mymod = type(__builtins__)('mymod') >>> for one_letter in 'abc': setattr(mymod, one_letter, []) >>> vars(mymod) {'__name__': 'mymod', '__doc__': None, '__package__': None, '__loader__': None, '__spec__': None, 'a': [], 'b': [], 'c': []}         

Sure enough, we’ve managed to do it!

By the way, remember how I mentioned, all the way back, that it would probably be best to use a dictionary, rather than create actual variables? Well, as you can see here, a module is actually just a fancy wrapper around… a dictionary.

This seemingly simple question raised all sorts of interesting Python functionality, none of which (I’m guessing) was ever intended by the person who asked the question. But I hope that this has given you a glimpse into the ways in which Python has implemented, and how a dynamic language allows us to play with our environment in ways that not only stretch our minds, but sometimes even the boundaries of good taste.

The post Playing with Python strings, lists, and variable names — or, a complex answer to a simple question appeared first on Reuven Lerner.

Planet Python