Stack Abuse: Python List Sorting with sorted() and sort()

In this article, we’ll examine multiple ways to sort lists in Python.

Python ships with two built-in methods for sorting lists and other iterable objects. The method chosen for a particular use-case often depends on whether we want to sort a list in-place or return a new version of the sorted list.

Assuming we want to sort a list in place, we can use the list.sort() method as follows:

>>> pets = ['Turtle', 'Cat', 'Fish', 'Dingo'] >>> pets.sort() >>> pets ['Cat', 'Dingo', 'Fish', 'Turtle'] 

By default, the list is sorted in ascending order. Notice how the original pets list is changed after the sort method is called on it. If we don’t want this to happen, we can use the built-in sorted() function to return a new sorted list while leaving the original list unchanged:

>>> pets = ['Turtle', 'Cat', 'Fish', 'Dingo'] >>> new_pets = sorted(pets) >>> new_pets ['Cat', 'Dingo', 'Fish', 'Turtle'] >>> pets ['Turtle', 'Cat', 'Fish', 'Dingo'] 

The reverse argument can be used to sort lists in descending order:

>>> pets = ['Turtle', 'Cat', 'Fish', 'Dingo'] >>> new_pets = sorted(pets, reverse=True) >>> new_pets ['Turtle', 'Fish', 'Dingo', 'Cat'] >>> pets.sort(reverse=True) >>> pets ['Turtle', 'Fish', 'Dingo', 'Cat'] 

However, there are scenarios where we might want to sort a list based on custom criteria that we define. For example, we may want to sort our pets list by the length of each entry. In that case Python offers the key argument, which accepts a user-defined function to specify the sorting criteria:

>>> pets = ['Turtle', 'Cat', 'Fish', 'Dingo'] >>> get_len(x): ...    return len(x) ... >>> new_pets = sorted(pets, key=get_len) >>> new_pets ['Cat', 'Fish', 'Dingo', 'Turtle'] >>> pets.sort(key=get_len) >>> pets ['Cat', 'Fish', 'Dingo', 'Turtle'] 

Now let’s consider a slightly more complex example. Here we have a list of dictionaries that contain data about a group of people, and we want to sort the list based the peoples’ ages, in descending order. To do this we will use both the key and reverse keyword arguments, as well as a Python lambda function. That way we can create the sorting function on the fly, instead of defining it beforehand:

>>> data = [ { 'name': 'Billy', 'age': 26, 'country': 'USA' }, { 'name': 'Timmy', 'age': 5, 'country': 'Australia' }, { 'name': 'Sally', 'age': 19, 'country': 'Costa Rica' }, { 'name': 'Tommy', 'age': 67, 'country': 'Serbia' } ] >>> new_data = sorted(data, key=lambda x: x['age'], reverse=True) >>> new_data [{'country': 'Serbia', 'age': 67, 'name': 'Tommy'}, {'country': 'USA', 'age': 26, 'name': 'Billy'}, {'country': 'Costa Rica', 'age': 19, 'name': 'Sally'}, {'country': 'Australia', 'age': 5, 'name': 'Timmy'}] >>> data.sort(key=lambda x: x['age'], reverse=True) >>> data [{'country': 'Serbia', 'age': 67, 'name': 'Tommy'}, {'country': 'USA', 'age': 26, 'name': 'Billy'}, {'country': 'Costa Rica', 'age': 19, 'name': 'Sally'}, {'country': 'Australia', 'age': 5, 'name': 'Timmy'}] 

Notice how the dictionaries startd in a seamingly random order and then ended up with the oldest people first and youngest people last in the list.

Using the sorting functions and lambdas in this way allows us to easily sort complex data structures, all in one line of code. And the order of the sort can be set to descending order by setting reverse=True.

About the Author

This article was written by Jacob Stopak, a software consultant and developer with a passion for helping others improve their lives through code. Jacob is the creator of Code Card – a convenient tool for developers to look up, copy, and paste common code snippets.


Planet Python

How To Install Linux, Apache, MariaDB, PHP (LAMP) stack on Debian 10

Introduction

A “LAMP” stack is a group of open-source software that is typically installed together to enable a server to host dynamic websites and web apps. This term is actually an acronym which represents the Linux operating system, with the Apache web server. The site data is stored in a MariaDB database, and dynamic content is processed by PHP.

Although this software stack typically includes MySQL as the database management system, some Linux distributions — including Debian — use MariaDB as a drop-in replacement for MySQL.

In this guide, we will install a LAMP stack on a Debian 10 server, using MariaDB as the database management system.

Prerequisites

In order to complete this tutorial, you will need to have a Debian 10 server with a non-root sudo-enabled user account and a basic firewall. This can be configured using our initial server setup guide for Debian 10.

Step 1 — Installing Apache and Updating the Firewall

The Apache web server is among the most popular web servers in the world. It’s well-documented and has been in wide use for much of the history of the web, which makes it a great default choice for hosting a website.

Install Apache using Debian’s package manager, APT:

  • sudo apt update
  • sudo apt install apache2

Since this is a sudo command, these operations are executed with root privileges. It will ask you for your regular user’s password to verify your intentions.

Once you’ve entered your password, apt will tell you which packages it plans to install and how much extra disk space they’ll take up. Press Y and hit ENTER to continue, and the installation will proceed.

Next, assuming that you have followed the initial server setup instructions by installing and enabling the UFW firewall, make sure that your firewall allows HTTP and HTTPS traffic.

When installed on Debian 10, UFW comes loaded with app profiles which you can use to tweak your firewall settings. View the full list of application profiles by running:

  • sudo ufw app list

The WWW profiles are used to manage ports used by web servers:

Output
Available applications: . . . WWW WWW Cache WWW Full WWW Secure . . .

If you inspect the WWW Full profile, it shows that it enables traffic to ports 80 and 443:

  • sudo ufw app info "WWW Full"
Output
Profile: WWW Full Title: Web Server (HTTP,HTTPS) Description: Web Server (HTTP,HTTPS) Ports: 80,443/tcp

Allow incoming HTTP and HTTPS traffic for this profile:

  • sudo ufw allow in "WWW Full"

You can do a spot check right away to verify that everything went as planned by visiting your server’s public IP address in your web browser:

http://your_server_ip 

You will see the default Debian 10 Apache web page, which is there for informational and testing purposes. It should look something like this:

Debian 10 Apache default

If you see this page, then your web server is now correctly installed and accessible through your firewall.

If you do not know what your server’s public IP address is, there are a number of ways you can find it. Usually, this is the address you use to connect to your server through SSH.

There are a few different ways to do this from the command line. First, you could use the iproute2 tools to get your IP address by typing this:

  • ip addr show eth0 | grep inet | awk '{ print $ 2; }' | sed 's/\/.*$ //'

This will give you two or three lines back. They are all correct addresses, but your computer may only be able to use one of them, so feel free to try each one.

An alternative method is to use the curl utility to contact an outside party to tell you how it sees your server. This is done by asking a specific server what your IP address is:

  • sudo apt install curl
  • curl http://icanhazip.com

Regardless of the method you use to get your IP address, type it into your web browser’s address bar to view the default Apache page.

Step 2 — Installing MariaDB

Now that you have a web server up and running, you need to install the database system to be able to store and manage data for your site.

In Debian 10, the metapackage mysql-server, which was traditionally used to install the MySQL server, was replaced by default-mysql-server. This metapackage references MariaDB, a community fork of the original MySQL server by Oracle, and it’s currently the default MySQL-compatible database server available on debian-based package manager repositories.

For longer term compatibility, however, it’s recommended that instead of using the metapackage you install MariaDB using the program’s actual package, mariadb-server.

To install this software, run:

  • sudo apt install mariadb-server

When the installation is finished, it’s recommended that you run a security script that comes pre-installed with MariaDB. This script will remove some insecure default settings and lock down access to your database system. Start the interactive script by running:

  • sudo mysql_secure_installation

This script will take you through a series of prompts where you can make some changes to your MariaDB setup. The first prompt will ask you to enter the current database root password. This is not to be confused with the system root. The database root user is an administrative user with full privileges over the database system. Because you just installed MariaDB and haven’t made any configuration changes yet, this password will be blank, so just press ENTER at the prompt.

The next prompt asks you whether you’d like to set up a database root password. Because MariaDB uses a special authentication method for the root user that is typically safer than using a password, you don’t need to set this now. Type N and then press ENTER.

From there, you can press Y and then ENTER to accept the defaults for all the subsequent questions. This will remove anonymous users and the test database, disable remote root login, and load these new rules so that MariaDB immediately respects the changes you have made.
When you’re finished, log in to the MariaDB console by typing:

  • sudo mariadb

This will connect to the MariaDB server as the administrative database user root, which is inferred by the use of sudo when running this command. You should see output like this:

Output
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 74 Server version: 10.3.15-MariaDB-1 Debian 10 Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>

Notice that you didn’t need to provide a password to connect as the root user. That works because the default authentication method for the administrative MariaDB user is unix_socket instead of password. Even though this might look like a security concern at first, it makes the database server more secure because the only users allowed to log in as the root MariaDB user are the system users with sudo privileges connecting from the console or through an application running with the same privileges. In practical terms, that means you won’t be able to use the administrative database root user to connect from your PHP application.

For increased security, it’s best to have dedicated user accounts with less expansive privileges set up for every database, especially if you plan on having multiple databases hosted on your server. To demonstrate such a setup, we’ll create a database named example_database and a user named example_user, but you can replace these names with different values.
To create a new database, run the following command from your MariaDB console:

  • CREATE DATABASE example_database;

Now you can create a new user and grant them full privileges on the custom database you’ve just created. The following command defines this user’s password as password, but you should replace this value with a secure password of your own choosing.

  • GRANT ALL ON example_database.* TO 'example_user'@'localhost' IDENTIFIED BY 'password' WITH GRANT OPTION;

This will give the example_user user full privileges over the example_database database, while preventing this user from creating or modifying other databases on your server.

Flush the privileges to ensure that they are saved and available in the current session:

  • FLUSH PRIVILEGES;

Following this, exit the MariaDB shell:

  • exit

You can test if the new user has the proper permissions by logging in to the MariaDB console again, this time using the custom user credentials:

  • mariadb -u example_user -p

Note the -p flag in this command, which will prompt you for the password used when creating the example_user user. After logging in to the MariaDB console, confirm that you have access to the example_database database:

  • SHOW DATABASES;

This will give you the following output:

Output
+--------------------+ | Database | +--------------------+ | example_database | | information_schema | +--------------------+ 2 rows in set (0.000 sec)

To exit the MariaDB shell, type:

  • exit

At this point, your database system is set up and you can move on to installing PHP, the final component of the LAMP stack.

Step 3 — Installing PHP

PHP is the component of your setup that will process code to display dynamic content. It can run scripts, connect to your MariaDB databases to get information, and hand the processed content over to your web server to display.

Once again, leverage the apt system to install PHP. In addition, include some helper packages which will ensure that PHP code can run under the Apache server and talk to your MariaDB database:

  • sudo apt install php libapache2-mod-php php-mysql

This should install PHP without any problems. We’ll test this in a moment.

In most cases, you will want to modify the way that Apache serves files. Currently, if a user requests a directory from the server, Apache will first look for a file called index.html. We want to tell the web server to prefer PHP files over others, so make Apache look for an index.php file first.

To do this, type the following command to open the dir.conf file in a text editor with root privileges:

  • sudo nano /etc/apache2/mods-enabled/dir.conf

It will look like this:

/etc/apache2/mods-enabled/dir.conf
<IfModule mod_dir.c>     DirectoryIndex index.html index.cgi index.pl index.php index.xhtml index.htm </IfModule> 

Move the PHP index file (highlighted above) to the first position after the DirectoryIndex specification, like this:

/etc/apache2/mods-enabled/dir.conf
<IfModule mod_dir.c>     DirectoryIndex index.php index.html index.cgi index.pl index.xhtml index.htm </IfModule> 

When you are finished, save and close the file. If you’re using nano, you can do that by pressing CTRL+X, then Y and ENTER to confirm.

Now reload Apache’s configuration with:

  • sudo systemctl reload apache2

You can check on the status of the apache2 service with systemctl status:

  • sudo systemctl status apache2
Sample Output
● apache2.service - The Apache HTTP Server Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2019-07-08 12:58:31 UTC; 8s ago Docs: https://httpd.apache.org/docs/2.4/ Process: 11948 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS) Main PID: 11954 (apache2) Tasks: 6 (limit: 4719) Memory: 11.5M CGroup: /system.slice/apache2.service ├─11954 /usr/sbin/apache2 -k start ├─11955 /usr/sbin/apache2 -k start ├─11956 /usr/sbin/apache2 -k start ├─11957 /usr/sbin/apache2 -k start ├─11958 /usr/sbin/apache2 -k start └─11959 /usr/sbin/apache2 -k start

At this point, your LAMP stack is fully operational, but before you can test your setup with a PHP script it’s best to set up a proper Apache Virtual Host to hold your website’s files and folders. We’ll do that in the next step.

Step 4 — Creating a Virtual Host for your Website

By default, Apache serves its content from a directory located at /var/www/html, using the configuration contained in /etc/apache2/sites-available/000-default.conf. Instead of modifying the default website configuration file, we are going to create a new virtual host for testing your PHP environment. Virtual hosts enable us to keep multiple websites hosted on a single Apache server.

Following that, you’ll create a directory structure within /var/www for an example website named your_domain.

Create the root web directory for your_domain as follows:

  • sudo mkdir /var/www/your_domain

Next, assign ownership of the directory with the $ USER environment variable, which should reference your current system user:

  • sudo chown -R $ USER:$ USER /var/www/your_domain

Then, open a new configuration file in Apache’s sites-available directory using your preferred command-line editor. Here, we’ll use nano:

  • sudo nano /etc/apache2/sites-available/your_domain.conf

This will create a new blank file. Paste in the following bare-bones configuration:

/etc/apache2/sites-available/your_domain
<VirtualHost *:80>     ServerName your_domain     ServerAlias www.your_domain      ServerAdmin webmaster@localhost     DocumentRoot /var/www/your_domain     ErrorLog $  {APACHE_LOG_DIR}/error.log     CustomLog $  {APACHE_LOG_DIR}/access.log combined </VirtualHost> 

With this VirtualHost configuration, we’re telling Apache to serve your_domain using /var/www/your_domain as the web root directory. If you’d like to test Apache without a domain name, you can remove or comment out the options ServerName and ServerAlias by adding a # character in the beginning of each option’s lines.

You can now use a2ensite to enable this virtual host:

  • sudo a2ensite your_domain

You might want to disable the default website that comes installed with Apache. This is required if you’re not using a custom domain name, because in this case Apache’s default configuration would overwrite your Virtual Host. To disable Apache’s default website, type:

  • sudo a2dissite 000-default

To make sure your configuration file doesn’t contain syntax errors, you can run:

  • sudo apache2ctl configtest

Finally, reload Apache so these changes take effect:

  • sudo systemctl reload apache2

Your new website is now active, but the web root /var/www/your_domain is still empty. In the next step, we’ll create a PHP script to test the new setup and confirm that PHP is correctly installed and configured on your server.

Step 5 — Testing PHP Processing on your Web Server

Now that you have a custom location to host your website’s files and folders, we’ll create a simple PHP test script to confirm that Apache is able to handle and process requests for PHP files.

Create a new file named info.php inside your custom web root folder:

  • nano /var/www/your_domain/info.php

This will open a blank file. Add the following text, which is valid PHP code, inside the file:

/var/www/your_domain/info.php
<?php phpinfo(); 

When you are finished, save and close the file.

Now you can test whether your web server is able to correctly display content generated by this PHP script. To try this out, visit this page in your web browser. You’ll need your server’s public IP address again.

The address you will want to visit is:

http://your_domain/info.php 

You should see a page similar to this:

Debian 10 default PHP info

This page provides some basic information about your server from the perspective of PHP. It is useful for debugging and to ensure that your settings are being applied correctly.

If you can see this page in your browser, then your PHP installation is working as expected.

After checking the relevant information about your PHP server through that page, it’s best to remove the file you created as it contains sensitive information about your PHP environment and your Debian server. You can use rm to do so:

  • sudo rm /var/www/your_domain/info.php

You can always recreate this page if you need to access the information again later.

Step 6 — Testing Database Connection from PHP (Optional)

If you want to test whether PHP is able to connect to MariaDB and execute database queries, you can create a test table with dummy data and query for its contents from a PHP script.

First, connect to the MariaDB console with the database user you created in Step 2 of this guide:

  • mariadb -u example_user -p

Create a table named todo_list. From the MariaDB console, run the following statement:

  • CREATE TABLE example_database.todo_list (
  • item_id INT AUTO_INCREMENT,
  • content VARCHAR(255),
  • PRIMARY KEY(item_id)
  • );

Now, insert a few rows of content in the test table. You might want to repeat the next command a few times, using different values:

  • INSERT INTO example_database.todo_list (content) VALUES ("My first important item");

To confirm that the data was successfully saved to your table, run:

  • SELECT * FROM example_database.todo_list;

You will see the following output:

Output
+---------+--------------------------+ | item_id | content | +---------+--------------------------+ | 1 | My first important item | | 2 | My second important item | | 3 | My third important item | | 4 | and this one more thing | +---------+--------------------------+ 4 rows in set (0.000 sec)

After confirming that you have valid data in your test table, you can exit the MariaDB console:

  • exit

Now you can create the PHP script that will connect to MariaDB and query for your content. Create a new PHP file in your custom web root directory using your preferred editor. We’ll use nano for that:

  • nano /var/www/your_domain/todo_list.php

The following PHP script connects to the MariaDB database and queries for the content of the todo_list table, exhibiting the results in a list. If there’s a problem with the database connection, it will throw an exception.
Copy this content into your todo_list.php script:

/var/www/your_domain/todo_list.php
<?php $  user = "example_user"; $  password = "password"; $  database = "example_database"; $  table = "todo_list";  try {   $  db = new PDO("mysql:host=localhost;dbname=$  database", $  user, $  password);   echo "<h2>TODO</h2><ol>";    foreach($  db->query("SELECT content FROM $  table") as $  row) {     echo "<li>" . $  row['content'] . "</li>";   }   echo "</ol>"; } catch (PDOException $  e) {     print "Error!: " . $  e->getMessage() . "<br/>";     die(); } 

Save and close the file when you’re done editing.

You can now access this page in your web browser by visiting the domain name or public IP address for your website, followed by /todo_list.php:

http://your_domain/todo_list.php 

You should see a page like this, showing the content you’ve inserted in your test table:

Example PHP todo list

That means your PHP environment is ready to connect and interact with your MariaDB server.

Conclusion

In this guide, we’ve built a flexible foundation for serving PHP websites and applications to your visitors, using Apache as web server and MariaDB as database system.

To further improve your current setup, you can install Composer for dependency and package management in PHP, and you can also install an OpenSSL certificate for your website using Let’s Encrypt.

DigitalOcean Community Tutorials

How To Install Linux, Nginx, MariaDB, PHP (LEMP stack) on Debian 10

Introduction

The LEMP software stack is a group of software that can be used to serve dynamic web pages and web applications. The name “LEMP” is an acronym that describes a Linux operating system, with an (E)Nginx web server. The backend data is stored in a MariaDB database and the dynamic processing is handled by PHP.

Although this software stack typically includes MySQL as the database management system, some Linux distributions — including Debian — use MariaDB as a drop-in replacement for MySQL.

In this guide, you’ll install a LEMP stack on a Debian 10 server using MariaDB as the database management system.

Prerequisites

To complete this guide, you will need access to a Debian 10 server. This server should have a regular user configured with sudo privileges and a firewall enabled with ufw. To set this up, you can follow our Initial Server Setup with Debian 10 guide.

Step 1 — Installing the Nginx Web Server

In order to serve web pages to your site visitors, we are going to employ Nginx, a popular web server which is well known for its overall performance and stability.

All of the software you will be using for this procedure will come directly from Debian’s default package repositories. This means you can use the apt package management suite to complete the installation.

Since this is the first time you’ll be using apt for this session, you should start off by updating your local package index. You can then install the server:

  • sudo apt update
  • sudo apt install nginx

On Debian 10, Nginx is configured to start running upon installation.

If you have the ufw firewall running, you will need to allow connections to Nginx. You should enable the most restrictive profile that will still allow the traffic you want. Since you haven’t configured SSL for your server yet, for now you only need to allow HTTP traffic on port 80.

You can enable this by typing:

  • sudo ufw allow 'Nginx HTTP'

You can verify the change by typing:

  • sudo ufw status

You should see HTTP traffic allowed in the displayed output:

Output
Status: active To Action From -- ------ ---- OpenSSH ALLOW Anywhere Nginx HTTP ALLOW Anywhere OpenSSH (v6) ALLOW Anywhere (v6) Nginx HTTP (v6) ALLOW Anywhere (v6)

Now, test if the server is up and running by accessing your server’s domain name or public IP address in your web browser. If you do not have a domain name pointed at your server and you do not know your server’s public IP address, you can find it by typing one of the following into your terminal:

  • ip addr show eth0 | grep inet | awk '{ print $ 2; }' | sed 's/\/.*$ //'

This will print out a few IP addresses. You can try each of them in turn in your web browser.

Type one of the addresses that you receive in your web browser. It should take you to Nginx’s default landing page:

http://your_domain_or_IP 

Nginx default page

If you see the above page, you have successfully installed Nginx.

Step 2 — Installing MariaDB

Now that you have a web server up and running, you need to install the database system to be able to store and manage data for your site.

In Debian 10, the metapackage mysql-server, which was traditionally used to install the MySQL server, was replaced by default-mysql-server. This metapackage references MariaDB, a community fork of the original MySQL server by Oracle, and it’s currently the default MySQL-compatible database server available on debian-based package manager repositories.

For longer term compatibility, however, it’s recommended that instead of using the metapackage you install MariaDB using the program’s actual package, mariadb-server.

To install this software, run:

  • sudo apt install mariadb-server

When the installation is finished, it’s recommended that you run a security script that comes pre-installed with MariaDB. This script will remove some insecure default settings and lock down access to your database system. Start the interactive script by running:

  • sudo mysql_secure_installation

This script will take you through a series of prompts where you can make some changes to your MariaDB setup. The first prompt will ask you to enter the current database root password. This is not to be confused with the system root. The database root user is an administrative user with full privileges over the database system. Because you just installed MariaDB and haven’t made any configuration changes yet, this password will be blank, so just press ENTER at the prompt.

The next prompt asks you whether you’d like to set up a database root password. Because MariaDB uses a special authentication method for the root user that is typically safer than using a password, you don’t need to set this now. Type N and then press ENTER.

From there, you can press Y and then ENTER to accept the defaults for all the subsequent questions. This will remove anonymous users and the test database, disable remote root login, and load these new rules so that MariaDB immediately respects the changes you have made.
When you’re finished, log in to the MariaDB console by typing:

  • sudo mariadb

This will connect to the MariaDB server as the administrative database user root, which is inferred by the use of sudo when running this command. You should see output like this:

Output
Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 74 Server version: 10.3.15-MariaDB-1 Debian 10 Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]>

Notice that you didn’t need to provide a password to connect as the root user. That works because the default authentication method for the administrative MariaDB user is unix_socket instead of password. Even though this might look like a security concern at first, it makes the database server more secure because the only users allowed to log in as the root MariaDB user are the system users with sudo privileges connecting from the console or through an application running with the same privileges. In practical terms, that means you won’t be able to use the administrative database root user to connect from your PHP application.

For increased security, it’s best to have dedicated user accounts with less expansive privileges set up for every database, especially if you plan on having multiple databases hosted on your server. To demonstrate such a setup, we’ll create a database named example_database and a user named example_user, but you can replace these names with different values.
To create a new database, run the following command from your MariaDB console:

  • CREATE DATABASE example_database;

Now you can create a new user and grant them full privileges on the custom database you’ve just created. The following command defines this user’s password as password, but you should replace this value with a secure password of your own choosing.

  • GRANT ALL ON example_database.* TO 'example_user'@'localhost' IDENTIFIED BY 'password' WITH GRANT OPTION;

This will give the example_user user full privileges over the example_database database, while preventing this user from creating or modifying other databases on your server.

Flush the privileges to ensure that they are saved and available in the current session:

  • FLUSH PRIVILEGES;

Following this, exit the MariaDB shell:

  • exit

You can test if the new user has the proper permissions by logging in to the MariaDB console again, this time using the custom user credentials:

  • mariadb -u example_user -p

Note the -p flag in this command, which will prompt you for the password used when creating the example_user user. After logging in to the MariaDB console, confirm that you have access to the example_database database:

  • SHOW DATABASES;

This will give you the following output:

Output
+--------------------+ | Database | +--------------------+ | example_database | | information_schema | +--------------------+ 2 rows in set (0.000 sec)

To exit the MariaDB shell, type:

  • exit

At this point, your database system is set up and you can move on to installing PHP, the final component of the LEMP stack.

Step 3 — Installing PHP for Processing

You have Nginx installed to serve your content and MySQL installed to store and manage your data. Now you can install PHP to process code and generate dynamic content for the web server.

While Apache embeds the PHP interpreter in each request, Nginx requires an external program to handle PHP processing and act as bridge between the PHP interpreter itself and the web server. This allows for a better overall performance in most PHP-based websites, but it requires additional configuration. You’ll need to install php-fpm, which stands for “PHP fastCGI process manager”, and tell Nginx to pass PHP requests to this software for processing. Additionally, you’ll need php-mysql, a PHP module that allows PHP to communicate with MySQL-based databases. Core PHP packages will automatically be installed as dependencies.

To install the php-fpm and php-mysql packages, run:

  • sudo apt install php-fpm php-mysql

You now have your PHP components installed. Next, you’ll configure Nginx to use them.

Step 4 — Configuring Nginx to Use the PHP Processor

When using the Nginx web server, server blocks (similar to virtual hosts in Apache) can be used to encapsulate configuration details and host more than one domain on a single server. In this guide, we’ll use your_domain as example domain name. To learn more about setting up a domain name with DigitalOcean, see our introduction to DigitalOcean DNS.

OnDebian 10, Nginx has one server block enabled by default and is configured to serve documents out of a directory at /var/www/html. While this works well for a single site, it can become difficult to manage if you are hosting multiple sites. Instead of modifying /var/www/html, let’s create a directory structure within /var/www for the your_domain website, leaving /var/www/html in place as the default directory to be served if a client request doesn’t match any other sites.

Create the root web directory for your_domain as follows:

  • sudo mkdir /var/www/your_domain

Next, assign ownership of the directory with the $ USER environment variable, which should reference your current system user:

  • sudo chown -R $ USER:$ USER /var/www/your_domain

Then, open a new configuration file in Nginx’s sites-available directory using your preferred command-line editor. Here, we’ll use nano:

  • sudo nano /etc/nginx/sites-available/your_domain

This will create a new blank file. Paste in the following bare-bones configuration:

/etc/nginx/sites-available/your_domain
server {     listen 80;     listen [::]:80;      root /var/www/your_domain;     index index.php index.html index.htm;      server_name your_domain;      location / {         try_files $  uri $  uri/ =404;     }      location ~ \.php$   {         include snippets/fastcgi-php.conf;         fastcgi_pass unix:/var/run/php/php7.3-fpm.sock;     } } 

This is a basic configuration that listens on port 80 and serves files from the web root you just created. It will only respond to requests to the host or IP address provided after server_name, and any files ending in .php will be processed by php-fpm before Nginx sends the results to the user.

When you’re done editing, save and close the file. If you used nano to create the file, do so by typing CTRL+X and then y and ENTER to confirm.

Activate your configuration by linking to the config file from Nginx’s sites-enabled directory:

  • sudo ln -s /etc/nginx/sites-available/your_domain /etc/nginx/sites-enabled/

This will tell Nginx to use the configuration next time it is reloaded. You can test your configuration for syntax errors by typing:

  • sudo nginx -t

If any errors are reported, go back to your configuration file to review its contents before continuing.

When you are ready, reload Nginx to make the changes:

  • sudo systemctl reload nginx

Next, you’ll create a file in your new web root directory to test out PHP processing.

Step 5 — Creating a PHP File to Test Configuration

Your LEMP stack should now be completely set up. You can test it to validate that Nginx can correctly hand .php files off to your PHP processor.

You can do this by creating a test PHP file in your document root. Open a new file called info.php within your document root in your text editor:

  • nano /var/www/your_domain/info.php

Type or paste the following lines into the new file. This is valid PHP code that will return information about your server:

/var/www/your_domain/info.php
<?php phpinfo(); 

When you are finished, save and close the file by typing CTRL+X and then y and ENTER to confirm.

You can now access this page in your web browser by visiting the domain name or public IP address you’ve set up in your Nginx configuration file, followed by /info.php:

http://your_domain/info.php 

You will see a web page containing detailed information about your server:

PHP page info

After checking the relevant information about your PHP server through that page, it’s best to remove the file you created as it contains sensitive information about your PHP environment and your Debian server. You can use rm to remove that file:

  • sudo rm /var/www/your_domain/info.php

You can always regenerate this file if you need it later. Next, we’ll test the database connection from the PHP side.

Step 6 — Testing Database Connection from PHP (Optional)

If you want to test if PHP is able to connect to MariaDB and execute database queries, you can create a test table with dummy data and query for its contents from a PHP script.

First, connect to the MariaDB console with the database user you created in Step 2 of this guide:

  • mariadb -u example_user -p

Create a table named todo_list. From the MariaDB console, run the following statement:

CREATE TABLE example_database.todo_list (     item_id INT AUTO_INCREMENT,     content VARCHAR(255),     PRIMARY KEY(item_id) ); 

Now, insert a few rows of content in the test table. You might want to repeat the next command a few times, using different values:

  • INSERT INTO example_database.todo_list (content) VALUES ("My first important item");

To confirm that the data was successfully saved to your table, run:

  • SELECT * FROM example_database.todo_list;

You will see the following output:

Output
+---------+--------------------------+ | item_id | content | +---------+--------------------------+ | 1 | My first important item | | 2 | My second important item | | 3 | My third important item | | 4 | and this one more thing | +---------+--------------------------+ 4 rows in set (0.000 sec)

After confirming that you have valid data in your test table, you can exit the MariaDB console:

  • exit

Now you can create the PHP script that will connect to MariaDB and query for your content. Create a new PHP file in your custom web root directory using your preferred editor. We’ll use nano for that:

  • nano /var/www/your_domain/todo_list.php

Add the following content to your PHP script:

<?php $  user = "example_user"; $  password = "password"; $  database = "example_database"; $  table = "todo_list";  try {   $  db = new PDO("mysql:host=localhost;dbname=$  database", $  user, $  password);   echo "<h2>TODO</h2><ol>";    foreach($  db->query("SELECT content FROM $  table") as $  row) {     echo "<li>" . $  row['content'] . "</li>";   }   echo "</ol>"; } catch (PDOException $  e) {     print "Error!: " . $  e->getMessage() . "<br/>";     die(); } 

Save and close the file when you’re done editing.

You can now access this page in your web browser by visiting the domain name or public IP address you’ve set up in your Nginx configuration file, followed by /todo_list.php:

http://your_domain/todo_list.php 

You should see a page like this, showing the content you’ve inserted in your test table:

Example PHP todo list

That means your PHP environment is ready to connect and interact with your MariaDB server.

Conclusion

In this guide, you’ve built a flexible foundation for serving PHP websites and applications to your visitors, using Nginx as web server. You’ve set up Nginx to handle PHP requests through php-fpm, and you also set up a MariaDB database to store your website’s data.

To further improve your current setup, you can install Composer for dependency and package management in PHP, and you can also set up an OpenSSL certificate for your website using Let’s Encrypt.

DigitalOcean Community Tutorials

How to Set Up a Prometheus, Grafana and Alertmanager Monitoring Stack on DigitalOcean Kubernetes

Introduction

Along with tracing and logging, monitoring and alerting are essential components of a Kubernetes observability stack. Setting up monitoring for your DigitalOcean Kubernetes cluster allows you to track your resource usage and analyze and debug application errors.

A monitoring system usually consists of a time-series database that houses metric data and a visualization layer. In addition, an alerting layer creates and manages alerts, handing them off to integrations and external services as necessary. Finally, one or more components generate or expose the metric data that will be stored, visualized, and processed for alerts by the stack.

One popular monitoring solution is the open-source Prometheus, Grafana, and Alertmanager stack, deployed alongside kube-state-metrics and node_exporter to expose cluster-level Kubernetes object metrics as well as machine-level metrics like CPU and memory usage.

Rolling out this monitoring stack on a Kubernetes cluster requires configuring individual components, manifests, Prometheus metrics, and Grafana dashboards, which can take some time. The DigitalOcean Kubernetes Cluster Monitoring Quickstart, released by the DigitalOcean Community Developer Education team, contains fully defined manifests for a Prometheus-Grafana-Alertmanager cluster monitoring stack, as well as a set of preconfigured alerts and Grafana dashboards. It can help you get up and running quickly, and forms a solid foundation from which to build your observability stack.

In this tutorial, we’ll deploy this preconfigured stack on DigitalOcean Kubernetes, access the Prometheus, Grafana, and Alertmanager interfaces, and describe how to customize it.

Prerequisites

Before you begin, you’ll need a DigitalOcean Kubernetes cluster available to you, and the following tools installed in your local development environment:

  • The kubectl command-line interface installed on your local machine and configured to connect to your cluster. You can read more about installing and configuring kubectl in its official documentation.
  • The git version control system installed on your local machine. To learn how to install git on Ubuntu 18.04, consult How To Install Git on Ubuntu 18.04.
  • The Coreutils base64 tool installed on your local machine. If you’re using a Linux machine, this will most likely already be installed. If you’re using OS X, you can use openssl base64, which comes installed by default.

Note: The Cluster Monitoring Quickstart has only been tested on DigitalOcean Kubernetes clusters. To use the Quickstart with other Kubernetes clusters, some modification to the manifest files may be necessary.

Step 1 — Cloning the GitHub Repository and Configuring Environment Variables

To start, clone the DigitalOcean Kubernetes Cluster Monitoring GitHub repository onto your local machine using git:

  • git clone git@github.com:do-community/doks-monitoring.git

Then, navigate into the repo:

  • cd doks-monitoring

You should see the following directory structure:

  • ls
Output
LICENSE README.md changes.txt manifest

The manifest directory contains Kubernetes manifests for all of the monitoring stack components, including Service Accounts, Deployments, StatefulSets, ConfigMaps, etc. To learn more about these manifest files and how to configure them, skip ahead to Configuring the Monitoring Stack.

If you just want to get things up and running, begin by setting the APP_INSTANCE_NAME and NAMESPACE environment variables, which will be used to configure a unique name for the stack’s components and configure the Namespace into which the stack will be deployed:

  • export APP_INSTANCE_NAME=sammy-cluster-monitoring
  • export NAMESPACE=default

In this tutorial, we set APP_INSTANCE_NAME to sammy-cluster-monitoring, which will prepend all of the monitoring stack Kubernetes object names. You should substitute in a unique descriptive prefix for your monitoring stack. We also set the Namespace to default. If you’d like to deploy the monitoring stack to a Namespace other than default, ensure that you first create it in your cluster:

  • kubectl create namespace "$ NAMESPACE"

You should see the following output:

Output
namespace/sammy created

In this case, the NAMESPACE environment variable was set to sammy. Throughout the rest of the tutorial we’ll assume that NAMESPACE has been set to default.

Now, use the base64 command to base64-encode a secure Grafana password. Be sure to substitute a password of your choosing for your_grafana_password:

  • export GRAFANA_GENERATED_PASSWORD="$ (echo -n 'your_grafana_password' | base64)"

If you’re using macOS, you can substitute the openssl base64 command which comes installed by default.

At this point, you’ve grabbed the stack’s Kubernetes manifests and configured the required environment variables, so you’re now ready to substitute the configured variables into the Kubernetes manifest files and create the stack in your Kubernetes cluster.

Step 2 — Creating the Monitoring Stack

The DigitalOcean Kubernetes Monitoring Quickstart repo contains manifests for the following monitoring, scraping, and visualization components:

  • Prometheus is a time series database and monitoring tool that works by polling metrics endpoints and scraping and processing the data exposed by these endpoints. It allows you to query this data using PromQL, a time series data query language. Prometheus will be deployed into the cluster as a StatefulSet with 2 replicas that uses Persistent Volumes with DigitalOcean Block Storage. In addition, a preconfigured set of Prometheus Alerts, Rules, and Jobs will be stored as a ConfigMap. To learn more about these, skip ahead to the Prometheus section of Configuring the Monitoring Stack.
  • Alertmanager, usually deployed alongside Prometheus, forms the alerting layer of the stack, handling alerts generated by Prometheus and deduplicating, grouping, and routing them to integrations like email or PagerDuty. Alertmanager will be installed as a StatefulSet with 2 replicas. To learn more about Alertmanager, consult Alerting from the Prometheus docs.
  • Grafana is a data visualization and analytics tool that allows you to build dashboards and graphs for your metrics data. Grafana will be installed as a StatefulSet with one replica. In addition, a preconfigured set of Dashboards generated by kubernetes-mixin will be stored as a ConfigMap.
  • kube-state-metrics is an add-on agent that listens to the Kubernetes API server and generates metrics about the state of Kubernetes objects like Deployments and Pods. These metrics are served as plaintext on HTTP endpoints and consumed by Prometheus. kube-state-metrics will be installed as an auto-scalable Deployment with one replica.
  • node-exporter, a Prometheus exporter that runs on cluster nodes and provides OS and hardware metrics like CPU and memory usage to Prometheus. These metrics are also served as plaintext on HTTP endpoints and consumed by Prometheus. node-exporter will be installed as a DaemonSet.

By default, along with scraping metrics generated by node-exporter, kube-state-metrics, and the other components listed above, Prometheus will be configured to scrape metrics from the following components:

  • kube-apiserver, the Kubernetes API server.
  • kubelet, the primary node agent that interacts with kube-apiserver to manage Pods and containers on a node.
  • cAdvisor, a node agent that discovers running containers and collects their CPU, memory, filesystem, and network usage metrics.

To learn more about configuring these components and Prometheus scraping jobs, skip ahead to Configuring the Monitoring Stack. We’ll now substitute the environment variables defined in the previous step into the repo’s manifest files, and concatenate the individual manifests into a single master file.

Begin by using awk and envsubst to fill in the APP_INSTANCE_NAME, NAMESPACE, and GRAFANA_GENERATED_PASSWORD variables in the repo’s manifest files. After substituting in the variable values, the files will be combined and saved into a master manifest file called sammy-cluster-monitoring_manifest.yaml.

  • awk 'FNR==1 {print "---"}{print}' manifest/* \
  • | envsubst '$ APP_INSTANCE_NAME $ NAMESPACE $ GRAFANA_GENERATED_PASSWORD' \
  • > "$ {APP_INSTANCE_NAME}_manifest.yaml"

You should consider storing this file in version control so that you can track changes to the monitoring stack and roll back to previous versions. If you do this, be sure to scrub the admin-password variable from the file so that you don’t check your Grafana password into version control.

Now that you’ve generated the master manifest file, use kubectl apply -f to apply the manifest and create the stack in the Namespace you configured:

  • kubectl apply -f "$ {APP_INSTANCE_NAME}_manifest.yaml" --namespace "$ {NAMESPACE}"

You should see output similar to the following:

Output
serviceaccount/alertmanager created configmap/sammy-cluster-monitoring-alertmanager-config created service/sammy-cluster-monitoring-alertmanager-operated created service/sammy-cluster-monitoring-alertmanager created . . . clusterrolebinding.rbac.authorization.k8s.io/prometheus created configmap/sammy-cluster-monitoring-prometheus-config created service/sammy-cluster-monitoring-prometheus created statefulset.apps/sammy-cluster-monitoring-prometheus created

You can track the stack’s deployment progress using kubectl get all. Once all of the stack components are RUNNING, you can access the preconfigured Grafana dashboards through the Grafana web interface.

Step 3 — Accessing Grafana and Exploring Metrics Data

The Grafana Service manifest exposes Grafana as a ClusterIP Service, which means that it’s only accessible via a cluster-internal IP address. To access Grafana outside of your Kubernetes cluster, you can either use kubectl patch to update the Service in-place to a public-facing type like NodePort or LoadBalancer, or kubectl port-forward to forward a local port to a Grafana Pod port. In this tutorial we’ll forward ports, so you can skip ahead to Forwarding a Local Port to Access the Grafana Service. The following section on exposing Grafana externally is included for reference purposes.

Exposing the Grafana Service using a Load Balancer (optional)

If you’d like to create a DigitalOcean Load Balancer for Grafana with an external public IP, use kubectl patch to update the existing Grafana Service in-place to the LoadBalancer Service type:

  • kubectl patch svc "$ APP_INSTANCE_NAME-grafana" \
  • --namespace "$ NAMESPACE" \
  • -p '{"spec": {"type": "LoadBalancer"}}'

The kubectl patch command allows you to update Kubernetes objects in-place to make changes without having to re-deploy the objects. You can also modify the master manifest file directly, adding a type: LoadBalancer parameter to the Grafana Service spec. To learn more about kubectl patch and Kubernetes Service types, you can consult the Update API Objects in Place Using kubectl patch and Services resources in the official Kubernetes docs.

After running the above command, you should see the following:

Output
service/sammy-cluster-monitoring-grafana patched

It may take several minutes to create the Load Balancer and assign it a public IP. You can track its progress using the following command with the -w flag to watch for changes:

  • kubectl get service "$ APP_INSTANCE_NAME-grafana" -w

Once the DigitalOcean Load Balancer has been created and assigned an external IP address, you can fetch its external IP using the following commands:

  • SERVICE_IP=$ (kubectl get svc $ APP_INSTANCE_NAME-grafana \
  • --namespace $ NAMESPACE \
  • --output jsonpath='{.status.loadBalancer.ingress[0].ip}')
  • echo "http://$ {SERVICE_IP}/"

You can now access the Grafana UI by navigating to http://SERVICE_IP/.

Forwarding a Local Port to Access the Grafana Service

If you don’t want to expose the Grafana Service externally, you can also forward local port 3000 into the cluster directly to a Grafana Pod using kubectl port-forward.

  • kubectl port-forward --namespace $ {NAMESPACE} $ {APP_INSTANCE_NAME}-grafana-0 3000

You should see the following output:

Output
Forwarding from 127.0.0.1:3000 -> 3000 Forwarding from [::1]:3000 -> 3000

This will forward local port 3000 to containerPort 3000 of the Grafana Pod sammy-cluster-monitoring-grafana-0. To learn more about forwarding ports into a Kubernetes cluster, consult Use Port Forwarding to Access Applications in a Cluster.

Visit http://localhost:3000 in your web browser. You should see the following Grafana login page:

Grafana Login Page

To log in, use the default username admin (if you haven’t modified the admin-user parameter), and the password you configured in Step 1.

You’ll be brought to the following Home Dashboard:

Grafana Home Page

In the left-hand navigation bar, select the Dashboards button, then click on Manage:

Grafana Dashboard Tab

You’ll be brought to the following dashboard management interface, which lists the dashboards configured in the dashboards-configmap.yaml manifest:

Grafana Dashboard List

These dashboards are generated by kubernetes-mixin, an open-source project that allows you to create a standardized set of cluster monitoring Grafana dashboards and Prometheus alerts. To learn more, consult the kubernetes-mixin GitHub repo.

Click in to the Kubernetes / Nodes dashboard, which visualizes CPU, memory, disk, and network usage for a given node:

Grafana Nodes Dashboard

Describing how to use these dashboards is outside of this tutorial’s scope, but you can consult the following resources to learn more:

In the next step, we’ll follow a similar process to connect to and explore the Prometheus monitoring system.

Step 4 — Accessing Prometheus and Alertmanager

To connect to the Prometheus Pods, we can use kubectl port-forward to forward a local port. If you’re done exploring Grafana, you can close the port-forward tunnel by hitting CTRL-C. Alternatively, you can open a new shell and create a new port-forward connection.

Begin by listing running Pods in the default namespace:

  • kubectl get pod -n default

You should see the following Pods:

Output
sammy-cluster-monitoring-alertmanager-0 1/1 Running 0 17m sammy-cluster-monitoring-alertmanager-1 1/1 Running 0 15m sammy-cluster-monitoring-grafana-0 1/1 Running 0 16m sammy-cluster-monitoring-kube-state-metrics-d68bb884-gmgxt 2/2 Running 0 16m sammy-cluster-monitoring-node-exporter-7hvb7 1/1 Running 0 16m sammy-cluster-monitoring-node-exporter-c2rvj 1/1 Running 0 16m sammy-cluster-monitoring-node-exporter-w8j74 1/1 Running 0 16m sammy-cluster-monitoring-prometheus-0 1/1 Running 0 16m sammy-cluster-monitoring-prometheus-1 1/1 Running 0 16m

We are going to forward local port 9090 to port 9090 of the sammy-cluster-monitoring-prometheus-0 Pod:

  • kubectl port-forward --namespace $ {NAMESPACE} sammy-cluster-monitoring-prometheus-0 9090

You should see the following output:

Output
Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090

This indicates that local port 9090 is being forwarded successfully to the Prometheus Pod.

Visit http://localhost:9090 in your web browser. You should see the following Prometheus Graph page:

Prometheus Graph Page

From here you can use PromQL, the Prometheus query language, to select and aggregate time series metrics stored in its database. To learn more about PromQL, consult Querying Prometheus from the official Prometheus docs.

In the Expression field, type kubelet_node_name and hit Execute. You should see a list of time series with the metric kubelet_node_name that reports the Nodes in your Kubernetes cluster. You can see which node generated the metric and which job scraped the metric in the metric labels:

Prometheus Query Results

Finally, in the top navigation bar, click on Status and then Targets to see the list of targets Prometheus has been configured to scrape. You should see a list of targets corresponding to the list of monitoring endpoints described at the beginning of Step 2.

To learn more about Prometheus and how to query your cluster metrics, consult the official Prometheus docs.

To connect to Alertmanager, which manages Alerts generated by Prometheus, we’ll follow a similar process to what we used to connect to Prometheus. . In general, you can explore Alertmanager Alerts by clicking into Alerts in the Prometheus top navigation bar.

To connect to the Alertmanager Pods, we will once again use kubectl port-forward to forward a local port. If you’re done exploring Prometheus, you can close the port-forward tunnel by hitting CTRL-Cor open a new shell to create a new connection. .

We are going to forward local port 9093 to port 9093 of the sammy-cluster-monitoring-alertmanager-0 Pod:

  • kubectl port-forward --namespace $ {NAMESPACE} sammy-cluster-monitoring-alertmanager-0 9093

You should see the following output:

Output
Forwarding from 127.0.0.1:9093 -> 9093 Forwarding from [::1]:9093 -> 9093

This indicates that local port 9093 is being forwarded successfully to an Alertmanager Pod.

Visit http://localhost:9093 in your web browser. You should see the following Alertmanager Alerts page:

Alertmanager Alerts Page

From here, you can explore firing alerts and optionally silencing them. To learn more about Alertmanager, consult the official Alertmanager documentation.

In the next step, you’ll learn how to optionally configure and scale some of the monitoring stack components.

Step 6 — Configuring the Monitoring Stack (optional)

The manifests included in the DigitalOcean Kubernetes Cluster Monitoring Quickstart repository can be modified to use different container images, different numbers of Pod replicas, different ports, and customized configuration files.

In this step, we’ll provide a high-level overview of each manifest’s purpose, and then demonstrate how to scale Prometheus up to 3 replicas by modifying the master manifest file.

To begin, navigate into the manifests subdirectory in the repo, and list the directory’s contents:

  • cd manifest
  • ls
Output
alertmanager-0serviceaccount.yaml alertmanager-configmap.yaml alertmanager-operated-service.yaml alertmanager-service.yaml . . . node-exporter-ds.yaml prometheus-0serviceaccount.yaml prometheus-configmap.yaml prometheus-service.yaml prometheus-statefulset.yaml

Here you’ll find manifests for the different monitoring stack components. To learn more about specific parameters in the manifests, click into the links and consult the comments included throughout the YAML files:

Alertmanager

Grafana

kube-state-metrics

node-exporter

Prometheus

  • prometheus-0serviceaccount.yaml: The Prometheus Service Account, ClusterRole and ClusterRoleBinding.
  • prometheus-configmap.yaml: A ConfigMap that contains three configuration files:

    • alerts.yaml: Contains a preconfigured set of alerts generated by kubernetes-mixin (which was also used to generate the Grafana dashboards). To learn more about configuring alerting rules, consult Alerting Rules from the Prometheus docs.
    • prometheus.yaml: Prometheus’s main configuration file. Prometheus has been preconfigured to scrape all the components listed at the beginning of Step 2. Configuring Prometheus goes beyond the scope of this article, but to learn more, you can consult Configuration from the official Prometheus docs.
    • rules.yaml: A set of Prometheus recording rules that enable Prometheus to compute frequently needed or computationally expensive expressions, and save their results as a new set of time series. These are also generated by kubernetes-mixin, and configuring them goes beyond the scope of this article. To learn more, you can consult Recording Rules from the official Prometheus documentation.
  • prometheus-service.yaml: The Service that exposes the Prometheus StatefulSet.

  • prometheus-statefulset.yaml: The Prometheus StatefulSet, configured with 2 replicas. This parameter can be scaled depending on your needs.

Example: Scaling Prometheus

To demonstrate how to modify the monitoring stack, we’ll scale the number of Prometheus replicas from 2 to 3.

Open the sammy-cluster-monitoring_manifest.yaml master manifest file using your editor of choice:

  • nano sammy-cluster-monitoring_manifest.yaml

Scroll down to the Prometheus StatefulSet section of the manifest:

Output
. . . apiVersion: apps/v1beta2 kind: StatefulSet metadata: name: sammy-cluster-monitoring-prometheus labels: &Labels k8s-app: prometheus app.kubernetes.io/name: sammy-cluster-monitoring app.kubernetes.io/component: prometheus spec: serviceName: "sammy-cluster-monitoring-prometheus" replicas: 2 podManagementPolicy: "Parallel" updateStrategy: type: "RollingUpdate" selector: matchLabels: *Labels template: metadata: labels: *Labels spec: . . .

Change the number of replicas from 2 to 3:

Output
. . . apiVersion: apps/v1beta2 kind: StatefulSet metadata: name: sammy-cluster-monitoring-prometheus labels: &Labels k8s-app: prometheus app.kubernetes.io/name: sammy-cluster-monitoring app.kubernetes.io/component: prometheus spec: serviceName: "sammy-cluster-monitoring-prometheus" replicas: 3 podManagementPolicy: "Parallel" updateStrategy: type: "RollingUpdate" selector: matchLabels: *Labels template: metadata: labels: *Labels spec: . . .

When you’re done, save and close the file.

Apply the changes using kubectl apply -f:

  • kubectl apply -f sammy-cluster-monitoring_manifest.yaml --namespace default

You can track progress using kubectl get pods. Using this same technique, you can update many of the Kubernetes parameters and much of the configuration for this observability stack.

Conclusion

In this tutorial, you installed a Prometheus, Grafana, and Alertmanager monitoring stack into your DigitalOcean Kubernetes cluster with a standard set of dashboards, Prometheus rules, and alerts.

You may also choose to deploy this monitoring stack using the Helm Kubernetes package manager. To learn more, consult How to Set Up DigitalOcean Kubernetes Cluster Monitoring with Helm and Prometheus. One additional way to get this stack up and running is to use the DigitalOcean Marketplace Kubernetes Monitoring Stack solution, currently in beta.

The DigitalOcean Kubernetes Cluster Monitoring Quickstart repository is heavily based on and modified from Google Cloud Platform’s click-to-deploy Prometheus solution. A full manifest of modifications and changes from the original repository can be found in the Quickstart repo’s changes.txt file.

DigitalOcean Community Tutorials

Stack Abuse: Python’s Bokeh Library for Interactive Data Visualization

Introduction

In this tutorial, we’re going to learn how to use Bokeh library in Python. Most of you would have heard of matplotlib, numpy, seaborn, etc. as they are very popular python libraries for graphics and visualizations. What distinguishes Bokeh from these libraries is that it allows dynamic visualization, which is supported by modern browsers (because it renders graphics using JS and HTML), and hence can be used for web applications with a very high level of interactivity.

Bokeh is available in R and Scala language as well; however, its Python counterpart is more commonly used than others.

Installation

The easiest way to install Boken using Python is through pip package manager. If you have pip installed in your system, run the following command to download and install Bokeh:

$   pip install bokeh 

Note: If you choose this method of installation, you need to have numpy installed in your system already

Another method to install Bokeh is through Anaconda distribution. Simply go to your terminal or command prompt and run this command:

$   conda install bokeh 

After completing this step, run the following command to ensure that your installation was successful:

$   bokeh --version 

If the above command runs successfully i.e. the version gets printed, then you can go ahead and use bokeh library in your programs.

Coding Exercises

In this part, we will be doing some hands-on examples by calling Bokeh library’s functions to create interactive visualizations. Let’s start by trying to make a square.

Note: Comments in the codes throughout this article are very important; they will not only explain the code but also convey other meaningful information. Furthermore, there might be ‘alternative’ or additional functionality that would be commented out, but you can try running it by uncommenting those lines.

Plotting Basic Shapes

Here we specify the x and y coordinates for points, which will be followed in sequence when the line is being drawn. The figure function instantiates a figure object, which stores the configurations of the graph you wish to plot. Here we can specify both the X range and Y range of the graph, which we set from 0 to 4, which covers the range of our data. The line method then draws a line between our coordinates, which is in the shape of a square.

from bokeh.io import output_file, output_notebook   from bokeh.plotting import figure, show  x = [1, 3, 3, 1, 1]   y = [1, 1, 3, 3, 1]  # Display the output in a separate HTML file  output_file('Square.html', title='Square in Bokeh')   #output_notebook() # Uncomment this line to use iPython notebook  square = figure(title='Square Shape',                plot_height=300, plot_width=300,              x_range=(0, 4), y_range=(0, 4))  # Draw a line using our data square.line(x, y)   #square.circle(x, y) # Uncomment this line to add a circle mark on each coordinate  # Show plot show(square)   

You may have noticed in the code that there is an alternative to the output_file function, which would instead show the result in a Jupyter notebook by using the output_notebook function. If you’d prefer to use a notebook then replace the output_file function with output_notebook in the code throughout this article.

When you run the above script, you should see the following square opening in a new tab of your default browser.

Output:

square plot

In the image above, you can see the tools on the right side (pan, box zoom, wheel zoom, save, reset, help – from top to bottom); these tools enable you to interact with the graph.

Another important thing which will come in handy is that after every call to the “show” function if you create a new “figure” object, a subsequent call to the “show” function with the new figure passed as an argument would generate an error. To resolve that error, run the following code:

from bokeh.plotting import reset_output  reset_output()   

The reset_output method resets the figure ID that the show function currently holds so that a new one can be assigned to it.

What we’ve done so far is rather basic, let’s now try to make multiple lines/map equations in a single graph. The most basic example for that would be to try and draw lines for the equations y = x, y = x^2, and y = x^3. So let’s see how we can make a graph to display them all at once using Bokeh:

from bokeh.plotting import figure, output_file, show  # Declare data for our three lines x = [1, 2, 3, 4, 5, 6]   x_square = [i**2 for i in x]   x_cube = [i**3 for i in x]  # Declare HTML file as output for when show is called output_file("Eqs.html")  lines = figure(title='Line Comparisons', x_range=(0, 8), y_range=(0,100),      x_axis_label='X-Axis', y_axis_label='Y-Axis')   lines.line(x, x, legend="y = x", line_width=3) # Line for the equation y=x   lines.square(x, x, legend="y = x", size=10) # Add square boxes on each point on the line  lines.line(x, x_square, legend="y = x^2", line_width=3) #Line for the equation y=x^2   lines.circle(x, x_square, legend="y = x^2", size=10) # Add circles to points since it partially overlaps with y=x  lines.line(x, x_cube, legend="y = x^3", line_width=3) # Line for the equation y=x^3   lines.square(x, x_cube, legend="y = x^2", size=10) # Add square boxes on each point of the line  # Display the graph show(lines)   

Output:

line comparisons graph

Before we continue to plot a few more graphics, let’s first learn a few cool tricks to make your graphics more interactive, as well as aesthetic. For that we’ll first of all learn about the different tools that the Bokeh Library uses apart from the ones that are displayed alongside (either on top or on the right side) the graph. The explanations will be provided in the comments of the code below:

# Use the same plot data as above x = [1, 2, 3, 4, 5, 6]   x_square = [i**2 for i in x]   x_cube = [i**3 for i in x]  #now let's make the necessary imports. Note that, in addition to the imports we made in the previous code, we'll be importing a few other things as well, which will be used to add more options in the 'toolset'.   # Same imports as before from bokeh.plotting import figure, output_file, show  # New imports to add more interactivity in our figures # Check out Bokeh's documentation for more tools (these are just two examples) from bokeh.models import HoverTool, BoxSelectTool  output_file("Eqs.html")  # Add the tools to this list tool_list = [HoverTool(), BoxSelectTool()]  # Use the tools keyword arg, otherwise the same lines = figure(title='Line Comparisons', x_range=(0, 8), y_range=(0, 100),      x_axis_label='X-Axis', y_axis_label='Y-Axis', tools=tool_list)  # The rest of the code below is the same as above lines.line(x, x, legend="y = x", line_width=3)   lines.square(x, x, legend="y = x", size=10)  lines.line(x, x_square, legend="y = x^2", line_width=3)   lines.circle(x, x_square, legend="y = x^2", size=10)  lines.line(x, x_cube, legend="y = x^3", line_width=3)   lines.square(x, x_cube, legend="y = x^2", size=10)  # Display the graph show(lines)   

Output:

extra tools

In the above picture, you can see the two extra options added to the previously available tools. You can now also hover over any data point and its details will be shown, and you can also select a certain group of data points to highlight them.

Handling Categorical Data with Bokeh

Next thing that we’ll learn to do using Bokeh library is handling categorical data. For that, we’ll try and make a bar chart first. To make it interesting, let’s try and create a chart which represents the number of world cups won by Argentina, Brazil, Spain, and Portugal. Sounds interesting? Let’s code it.

from bokeh.io import show, output_file   from bokeh. plotting import figure  output_file("cups.html")  # List of teams to be included in the chart. Add or # remove teams (and their World Cups won below) to # see how it affects the chart teams = ['Argentina', 'Brazil', 'Spain', 'Portugal']  # Activity: We experimented with the Hover Tool and the # Box Select tool in the previous example, try to # include those tools in this graph  # Number of world cups that the team has won wc_won = [5, 3, 4, 2]  # Setting toolbar_location=None and tools="" essentially # hides the toolbar from the graph barchart = figure(x_range=teams, plot_height=250, title="WC Counts",              toolbar_location=None, tools="")  barchart.vbar(x=teams, top=wc_won, width=0.5)  # Acitivity: Play with the width variable and see what # happens. In particular, try to set a value above 1 for # it   barchart.xgrid.grid_line_color = 'red'   barchart.y_range.start = 0  show(barchart)   

Output:

World cup count graph

Do you notice something in the graph above? It’s quite simple, and unimpressive, no? Let’s make some changes in the above code, and make it a bit more colorful and aesthetic. Bokeh has a lot of options to help us with that. Let’s see what we can do with it:

# Mostly the same code as above, except with a few # additions to add more color to our currently dry graph  from bokeh.io import show, output_file   from bokeh.plotting import figure  # New imports below from bokeh.models import ColumnDataSource  # A was added 4 to the end of Spectral because we have 4 # teams. If you had more or less you would have used that # number instead from bokeh.palettes import Spectral4  from bokeh.transform import factor_cmap  output_file("cups.html")  teams = ['Argentina', 'Brazil', 'Spain', 'Portugal']   wc_won = [5, 3, 4, 2]  source = ColumnDataSource(data=dict(teams=teams, wc_won=wc_won, color=Spectral4))  barchart = figure(x_range=teams, y_range=(0,8), plot_height=250, title="World Cups Won",              toolbar_location=None, tools="")  barchart.vbar(x='teams', top='wc_won', width=0.5, color='color', legend='teams', source=source)   # Here we change the position of the legend in the graph # Normally it is displayed as a vertical list on the top # right. These settings change that to a horizontal list # instead, and display it at the top center of the graph barchart.legend.orientation = "horizontal"   barchart.legend.location = "top_center"  show(barchart)   

Output:

improved World Cup count graph

Evidently, the new graph looks a lot better than before, with added interactivity.

Before concluding this article, I’d like to let you all know that this was just a glimpse of the functionality that Bokeh offers. There are tons of other cool things that you can do with it, and you should try them out by referring to Bokeh’s documentation and following the available examples.

Conclusion

To sum it up, in this tutorial we learned about the Bokeh library’s Python variant. We saw how to download and install it using the pip or anaconda distribution. We used Bokeh library programs to make interactive and dynamic visualizations of different types and using different data types as well. We also learned, by seeing practical examples, the reason why Bokeh is needed even though there are other more popular visualization libraries like matplotlib and Seaborn available. In short, Bokeh is very resourceful and can pretty much do all kinds of interactive visualizations that you may want.

Planet Python