Indexed by Google: Pages Checker on Python

To check if a webpage has been indexed by Google, you can manually search for site:<webpage> in a Google Search, for example: site:https://www.shellhacks.com/indexed-by-google-pages-checker-on-python.

If the page is indexed by Google, you will see the URL in the result of the Google Search, otherwise you will see the message as follows:

Your search – site:https://www.shellhacks.com/indexed-by-google-pages-checker-on-python – did not match any documents.

If you need to check multiple pages, it will be much more efficient to create a Python script for checking the pages indexed by Google.

This note shows an example of the minimal Python script for checking the pages indexed by Google, that can be used as a base and improved according to your needs. (more…)

PIP: Install From Private PyPi Repository

By default pip installs packages from a public PyPi repository but can also be configured to install them from the private repositories, like Nexus or Artifactory.

In this note i will show how to configure pip to install packages from the private repositories.

I will also show how to define username and password in pip for the private repositories that require authentication and how to troubleshoot the SSL related issues. (more…)

Pip: Install Requirements – Exclude Specific Packages

The requirements.txt file that contains the Python application dependencies is usually generated by developers using the pip freeze > requirements.txt command.

If you install dependencies with the pip install -r requirements.txt command as a part of some automated pipeline or a Dockerfile and you have an issue with some of the dependencies, you may wonder how to exclude and not to install the packages that cause the issue.

In this note i will show how to exclude the specific packages while installing the Python dependencies using the pip install -r requirements.txt command. (more…)

Pip Install – SSL Error: Certificate_Verify_Failed

If you are trying to install some Python package using the pip install command and pip fails to verify the SSL certificate, you may receive the error as follows:

Could not fetch URL https://pypi.org/…/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host=’pypi.org’, port=443): Max retries exceeded with url: /…/ (Caused by SSLError(SSLCertVerificationError(1, ‘[ SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: … – skipping

If there is a problem with confirming the SSL certificate of a repository, you can add it as a --trusted-host that will make pip ignore the SSL certificate check for this repository. (more…)

Python: Virtualenv – Install | Create | (De)activate

virtualenv serves for creating isolated Python environments.

It creates a folder with a copy of Python interpreter and a copy of the pip library which will be used to install other packages in this virtual environment without affecting other projects or system-wide libraries.

virtualenv makes it easier to work on more than one project at a time without introducing conflicts in their dependencies.

From this article you will find out how to install, create, activate and deactivate virtualenv. (more…)