How can I optimize Python code performance using profiling tools?
Open
15
I'm working on a Python application and running into an issue with Python concurrency. Here's the problematic code:
# Current implementation
class DataProcessor:
def __init__(self):
self.data = []
def process_large_file(self, filename):
with open(filename, 'r') as f:
self.data = f.readlines() # Memory issue with large files
return self.process_data()
The error message I'm getting is: "MemoryError: Unable to allocate array with shape and data type"
What I've tried so far:
- Used pdb debugger to step through the code
- Added logging statements to trace execution
- Checked Python documentation and PEPs
- Tested with different Python versions
- Reviewed similar issues on GitHub and Stack Overflow
Environment information:
- Python version: 3.11.0
- Operating system: Ubuntu 22.04
- Virtual environment: venv (activated)
- Relevant packages: django, djangorestframework, celery, redis
Any insights or alternative approaches would be very helpful. Thanks!
A
Asked by
azzani
Bronze
•
51 rep
1 Answer
17
Python decorators with arguments require a three-level nested function. Here's the proper implementation:
import functools
# Decorator with arguments
def retry(max_attempts=3, delay=1):
def decorator(func):
@functools.wraps(func) # Preserves function metadata
def wrapper(*args, **kwargs):
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == max_attempts - 1:
raise e
time.sleep(delay)
return wrapper
return decorator
# Usage
@retry(max_attempts=5, delay=2)
def unreliable_function():
# Function that might fail
pass
Class-based decorator (alternative approach):
class Retry:
def __init__(self, max_attempts=3, delay=1):
self.max_attempts = max_attempts
self.delay = delay
def __call__(self, func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(self.max_attempts):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == self.max_attempts - 1:
raise e
time.sleep(self.delay)
return wrapper
# Usage
@Retry(max_attempts=5, delay=2)
def another_function():
pass
J
Bronze
•
60 rep
Your Answer
You need to be logged in to answer questions.
Log In to Answer