Lab 1.2: Python Virtual Environments and Package Management

Overview

Installing Python packages globally on a system can lead to version conflicts, dependency issues, and broken production environments. Virtual environments solve this by creating isolated Python installations for each project.

In this lab, you'll learn to:

Create and activate virtual environments
Install and manage packages with pip
Export and recreate environments for reproducibility
Use virtual environment interpreters without activation
Configure scripts to run in specific environments

Prerequisites:

Completed Lab 1.1 (Python Basics)
SSH access to a Linux VM
Python 3.3+ installed

Part 1: Understanding the Problem

Step 1: Check Global Python Environment

# See where Python is installed
which python

# See globally installed packages
pip list

Step 2: The Global Install Problem

What happens when you install packages globally:

# Install a package globally (DON'T do this in production!)
pip install requests

Problems with global installs:

Project A needs requests==2.28.0
Project B needs requests==2.31.0
You can only have ONE version globally
System tools might break if you upgrade their dependencies
No reproducibility - different servers have different packages

The solution: Virtual environments!

Part 2: Create Your First Virtual Environment

Step 1: Create a Project Directory

# Create a project folder
mkdir ~/my-api-project
cd ~/my-api-project

Step 2: Create a Virtual Environment

# Create a venv named 'venv'
python -m venv venv

What this does:

Creates a venv/ directory
Copies Python interpreter into venv/bin/python
Creates isolated site-packages/ directory for packages
Includes pip and setuptools

Step 3: Explore the Virtual Environment

# See what was created
ls -la venv/

# Check the structure
tree venv/ -L 2  # If tree is installed, otherwise use ls -R

Key directories:

venv/bin/ - Python interpreter and activation scripts
venv/lib/python3.x/site-packages/ - Where packages get installed
venv/include/ - C header files for building packages

Part 3: Activate and Use the Virtual Environment

Step 1: Check Python Location (Before Activation)

# Before activation - using system Python
which python
python --version

Step 2: Activate the Virtual Environment

# Activate (notice the dot or 'source' command)
source venv/bin/activate

Note on Powershell

You will need to give the current shell (no system wide changes) permissions to execute

Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
.\venv\Scripts\Activate.ps1
(venv) PS C:\Users\blanc>

You'll see your prompt change:

(venv) user@hostname:~/my-api-project$

The (venv) prefix shows you're in the virtual environment!

Step 3: Check Python Location (After Activation)

# After activation - using venv Python
which python
# Output: /home/user/my-api-project/venv/bin/python

python --version

Notice: The Python path changed! You're now using the isolated interpreter.

Step 4: Check Installed Packages

# List packages in this venv
pip list

Expected output:

Package    Version
---------- -------
pip        XX.X.X
setuptools XX.X.X

Only base packages! Nothing from the global environment.

Part 4: Install Packages and Explore pip

Step 1: Install a Package

# Install the requests library
pip install requests

Step 2: Verify Installation

# List all packages
pip list

# Show details about a specific package
pip show requests

Output of pip show requests:

Name: requests
Version: 2.31.0
Location: /home/user/my-api-project/venv/lib/python3.x/site-packages
Requires: charset-normalizer, idna, urllib3, certifi

Notice the Location - it's in YOUR venv, not the global site-packages!

Step 3: Test the Package

# Quick test
python -c "import requests; print(requests.__version__)"

Part 5: Export Environment for Reproducibility

Step 1: Understanding pip list vs pip freeze

# Shows all installed packages with details
pip list

# Shows only what you explicitly installed (and their dependencies)
pip freeze

Key difference:

pip list - human-readable, shows everything
pip freeze - machine-readable, perfect for requirements files

Step 2: Create a Requirements File

# Export all installed packages
pip freeze > requirements.txt

# View the file
cat requirements.txt

Expected output:

certifi==2023.11.17
charset-normalizer==3.3.2
idna==3.6
requests==2.31.0
urllib3==2.1.0

This file lists exact versions - critical for reproducibility!

Step 3: Install More Packages

# Install additional packages
pip install python-dotenv

# Update requirements file
pip freeze > requirements.txt

# Check what changed
cat requirements.txt

Part 6: Recreate the Environment

Now let's prove we can recreate this exact environment on another machine (or project).

Step 1: Deactivate Current Environment

# Exit the virtual environment
deactivate

Notice: Your prompt returns to normal (no more (venv) prefix).

# Verify you're back to system Python
which python

Step 2: Create a New Virtual Environment

# Create a second project
mkdir ~/my-api-project-v2
cd ~/my-api-project-v2

# Create new venv
python -m venv venv

# Activate it
source venv/bin/activate

Step 3: Verify It's Empty

# Should only have pip and setuptools
pip list

Step 4: Install from Requirements File

# Copy requirements file from first project
cp ~/my-api-project/requirements.txt .

# Install everything at once
pip install -r requirements.txt

Watch it install all the packages!

Step 5: Verify Identical Environment

# Compare
pip freeze

# Should match the original requirements.txt exactly!
diff <(pip freeze) requirements.txt

No output from diff = identical environments!

Part 7: Using the Interpreter Without Activation

Here's a powerful technique: You don't actually need to activate a venv to use it!

Step 1: Deactivate (if activated)

deactivate

Step 2: Use the venv Python Directly

# Run Python from the venv without activating
~/my-api-project/venv/bin/python --version

# Test that packages work
~/my-api-project/venv/bin/python -c "import requests; print(requests.__version__)"

It works! The venv's Python knows to look in its own site-packages.

Step 3: Create a Test Script

cd ~/my-api-project

Create test_requests.py:

import requests

response = requests.get('https://api.github.com')
print(f"Status Code: {response.status_code}")
print(f"Using requests version: {requests.__version__}")

Step 4: Run Script with venv Python (No Activation!)

# Run with venv interpreter directly
./venv/bin/python test_requests.py

Why this matters:

Cron jobs can't activate environments
CI/CD scripts can use direct paths
System services need explicit interpreter paths

Part 8: Using Shebangs for Direct Execution

Step 1: Create a Script with a Shebang

Create api_checker.py: (replace ec2-user if needed)

#!/home/ec2-user/my-api-project/venv/bin/python

import requests
import sys

def check_api(url):
    try:
        response = requests.get(url, timeout=5)
        print(f"✓ {url} - Status: {response.status_code}")
        return response.status_code
    except Exception as e:
        print(f"✗ {url} - Error: {e}")
        return None

if __name__ == "__main__":
    urls = [
        "https://api.github.com",
        "https://httpbin.org/get",
    ]
    
    for url in urls:
        check_api(url)

Step 2: Make It Executable

# Add execute permission
chmod +x api_checker.py

Step 3: Run Directly (No python command needed!)

# Run it like a binary
./api_checker.py

What happened:

Linux reads the shebang line (#!/path/to/venv/bin/python)
Uses that interpreter to run the script
That interpreter has access to the venv packages
No activation needed!

Step 4: Verify It's Using the venv

Add these lines to the end of api_checker.py:

# Debug: Show which Python we're using
import sys
print(f"\nPython interpreter: {sys.executable}")

Then run it:

./api_checker.py

You'll see it's using the venv Python!

Part 9: Real-World Use Cases

Use Case 1: Docker Images

Docker containers use your requirements.txt file to install dependencies:

When building Docker images, you copy your requirements.txt into the container and install packages from it. The container itself provides isolation, so you don't need a virtual environment inside Docker. Your requirements.txt ensures every container has the exact same package versions.

Use Case 2: CI/CD Pipelines

GitHub Actions, Jenkins, GitLab CI all use requirements.txt:

CI/CD pipelines create fresh environments for every build. They typically:

Set up Python
Create a virtual environment
Install dependencies from your requirements.txt
Run tests or build artifacts

This ensures your tests run in a clean, reproducible environment every time.

Use Case 3: Multiple Python Versions

You can create virtual environments from different Python versions:

# Different projects, different Python versions (FOR REFERENCE - don't run!)
python3.9 -m venv ~/project-a/venv
python3.11 -m venv ~/project-b/venv
python3.12 -m venv ~/project-c/venv

Note: You likely only have one Python version installed right now. This example is just to show that if you had multiple Python versions on your system, you could create separate venvs for each. This is common in production environments where you need to maintain legacy applications alongside newer ones.

Part 10: Best Practices

Always Use requirements.txt

# Development
pip install requests flask pytest black

# Freeze it
pip freeze > requirements.txt

# Commit to git
git add requirements.txt
git commit -m "Add dependencies"

Add venv to .gitignore

Create .gitignore:

# Virtual environments
venv/
.venv/
env/
ENV/

# Python cache
__pycache__/
*.pyc
*.pyo
*.pyd

# IDEs
.vscode/
.idea/

Never commit virtual environments to git! They're large, binary files, and platform-specific.

Use pip freeze --local

# Only show packages installed in this venv (excludes global)
pip freeze --local > requirements.txt

Name Your venv Consistently

Common conventions:

venv/ - most common
.venv/ - hidden folder
env/ - shorter
virtualenv/ - explicit

Pick one and stick with it across all projects!

Part 11: Hands-On Challenge

Your task: Create an RSS feed reader environment

Create a new project: ~/rss-reader
Create and activate a venv
Install these packages: requests, beautifulsoup4, lxml
Find a website with an RSS feed (examples: news sites, blogs, tech sites)
Check the site's robots.txt file to verify scraping is allowed
Create a script that fetches the RSS feed and displays article titles
Export requirements.txt
Create a second environment and reproduce it
Make the script executable with a shebang

Important: Always check robots.txt before scraping:

Look for User-agent: * rules
Respect Disallow: directives
Check for Crawl-delay: requirements

Example RSS feeds to try:

https://news.ycombinator.com/rss
https://www.reddit.com/r/python/.rss
Any blog with /feed or /rss in the URL

Hints:

import requests
from bs4 import BeautifulSoup

# Fetch RSS feed
response = requests.get('YOUR_RSS_FEED_URL')
soup = BeautifulSoup(response.text, 'xml')  # Note: 'xml' not 'lxml'

# Find all items
items = soup.find_all('item')
for item in items:
    title = item.find('title').text
    link = item.find('link').text
    print(f"{title}\n{link}\n")

Success Criteria

You've completed this lab when you can:

Create a virtual environment with python -m venv
Activate and deactivate environments
Install packages with pip
Use pip list, pip show, and pip freeze
Export dependencies to requirements.txt
Recreate an environment from requirements.txt
Run Python scripts using the venv interpreter without activation
Create executable scripts with venv shebangs
Understand when and why to use virtual environments

Key Takeaways

What you learned:

Isolation - Each project has its own dependencies
Reproducibility - requirements.txt ensures consistent environments
Portability - Same environment on dev, staging, and production
Version control - Multiple versions of the same package across projects
Professional workflow - How real Python projects are managed