Python Multiprocessing: A Complete Guide

Python Multiprocessing

Introduction
In the age of multicore processors, leveraging the power of multiprocessing is essential for building efficient and fast applications. Python offers the built-in multiprocessing module, which provides tools for parallelizing tasks. This article explores multiprocessing in Python, its importance, and advanced features like queues, locks, and pools.

Download New Real Time Projects :-Click here


What is Multiprocessing?

Multiprocessing refers to running multiple processes simultaneously. It uses multiple CPU cores to enhance performance and efficiently manage tasks.

Key Features:

  • Tasks run independently and simultaneously.
  • Shared access to memory and peripherals.
  • Allocation of tasks to different CPUs by the operating system.

Example Analogy

Imagine a single chef handling all kitchen tasks. Without help, they switch between tasks like chopping, cooking, and cleaning. But if there are multiple chefs, each can work on specific tasks, ensuring faster and smoother operations.


Why Use Multiprocessing?

Without multiprocessing, systems use a single CPU core, leading to slower task execution due to frequent interruptions. Multiprocessing:

  • Increases efficiency.
  • Reduces execution time.
  • Simplifies tracking and managing tasks.

Multiprocessing in Python

The multiprocessing module in Python offers a user-friendly API for making use of several cores. It makes it simple for developers to create parallel programs.


Basic Multiprocessing Example

from multiprocessing import Process

def greet():
    print('Hello! Welcome to Python Multiprocessing!')

if __name__ == '__main__':
    p = Process(target=greet)
    p.start()
    p.join()

Output:

Hello! Welcome to Python Multiprocessing!

Explanation:

  1. Process: Creates a new process.
  2. start(): Starts the process.
  3. join(): Waits for the process to complete.

Multiprocessing with Arguments

import multiprocessing

def calculate_square(n):
    print(f"The square of {n} is {n * n}")

def calculate_cube(n):
    print(f"The cube of {n} is {n * n * n}")

if __name__ == "__main__":
    process1 = multiprocessing.Process(target=calculate_square, args=(4,))
    process2 = multiprocessing.Process(target=calculate_cube, args=(4,))

    process1.start()
    process2.start()

    process1.join()
    process2.join()

    print("Both processes are completed.")

Output:

The square of 4 is 16  
The cube of 4 is 64  
Both processes are completed.

Key Points:

  • args: Passes arguments to the target function.
  • Processes run concurrently, reducing overall execution time.

Advanced Multiprocessing Features

1. Queue Class

Used to share data between processes.

from multiprocessing import Queue

queue = Queue()

# Adding items to the queue
for item in ['Apple', 'Orange', 'Banana']:
    queue.put(item)

# Removing items from the queue
while not queue.empty():
    print(queue.get())

Output:

Apple  
Orange  
Banana

2. Lock Class

Ensures only one process accesses critical sections of code at a time.

from multiprocessing import Lock, Process

def print_message(lock, message):
    with lock:
        print(message)

if __name__ == "__main__":
    lock = Lock()
    messages = ["Task 1", "Task 2", "Task 3"]
    processes = [Process(target=print_message, args=(lock, msg)) for msg in messages]

    for p in processes:
        p.start()
    for p in processes:
        p.join()

Output:

Task 1  
Task 2  
Task 3

3. Pool Class

Used for parallel execution of functions.

Example 1: Distributing tasks

from multiprocessing import Pool

def square(n):
    return n * n

if __name__ == "__main__":
    with Pool(4) as pool:
        results = pool.map(square, [1, 2, 3, 4])
    print(results)

Output:

[1, 4, 9, 16]

Example 2: Simulating tasks

from multiprocessing import Pool
import time

def task(name):
    print(f"Task {name} started.")
    time.sleep(2)
    print(f"Task {name} completed.")

if __name__ == "__main__":
    with Pool(2) as pool:
        pool.map(task, ['A', 'B', 'C', 'D'])

Output:

Task A started.  
Task B started.  
Task A completed.  
Task B completed.  
Task C started.  
Task D started.  
Task C completed.  
Task D completed.

Common Multiprocessing Functions

FunctionDescription
cpu_count()Returns the number of CPUs available.
start()Starts a process.
join()Waits for a process to complete.
Queue()Creates a queue for data sharing.
Lock()Prevents concurrent access to shared resources.
Pool()Manages multiple processes for parallel execution.

PHP PROJECT:- CLICK HERE

INTERVIEW QUESTION:-CLICK HERE

Complete Advance AI topics:- CLICK HERE

Complete Python Course with Advance topics:- CLICK HERE


  • python multiprocessing pool
  • python multiprocessing example
  • python multiprocessing vs multithreading
  • python multiprocessing pool example
  • python multiprocessing queue
  • python multiprocessing tutorial
  • python multiprocessing return value
  • python multiprocessing for loop
  • python multithreading
  • python threading
See also  Python Arrays: A Comprehensive Guide

Post Comment