10 Powerful Python Tricks Every Data Scientist Should Know

ByJordan Brown

Python is an incredibly versatile and expressive language that‘s hugely popular in the data science community. While you can do a lot with the core language and built-in libraries, Python really shines when you take advantage of its more advanced features and syntax.

Knowing some handy Python "tricks" – creative ways to write more concise, readable, and efficient code – can really step up your data science game. Not only will you get more done in less time, you‘ll also be able to tackle more complex problems and build more sophisticated solutions.

In this post, we‘ll dive into 10 of the most powerful Python tricks that are especially valuable for data science. These span the data science workflow from data wrangling to machine learning, visualization and more.

Whether you‘re a Python newbie or a seasoned pro, there‘s sure to be something here that will make you say "Wow, I didn‘t know you could do that!" Let‘s get started.

1. Unpacking variables

One of the most common (and most annoying) things we have to do as data scientists is extract individual variables from lists or tuples. For example, say you have a tuple with latitude and longitude:

coordinates = (42.35, -71.08)

To extract latitude and longitude, you could do this:

latitude = coordinates[0]
longitude = coordinates[1]

But a much cleaner way is to use unpacking:

latitude, longitude = coordinates

This works with any list or tuple, making it an incredibly versatile trick. You can even use it to swap variable values without needing a temporary variable:

a = 1
b = 2
a, b = b, a

How cool is that?

2. List comprehensions

List comprehensions offer a compact way to create lists based on existing lists. Let‘s say you have a list of numbers and want to create a new list containing only the even numbers. Here‘s the traditional way:

numbers = [1, 2, 3, 4, 5]
even_numbers = []

for num in numbers:
    if num % 2 == 0:
        even_numbers.append(num)

And here‘s the list comprehension way:

numbers = [1, 2, 3, 4, 5]
even_numbers = [num for num in numbers if num % 2 == 0]

So much more concise and readable! List comprehensions can include multiple for clauses and if conditions, allowing you to create quite sophisticated lists in a single line of code. Whenever you find yourself writing a for loop to create a list, pause to see if you could use a list comprehension instead.

3. Lambda functions

Lambda functions are small, anonymous functions that can have any number of arguments but only one expression. They‘re handy for one-off operations where you don‘t want to bother defining a whole separate function.

For example, say you have a list of names and want to sort them by last name:

names = ["Alan Turing", "Ada Lovelace", "Claude Shannon", "Grace Hopper"]

names.sort(key=lambda name: name.split()[-1])

Here the lambda function takes a name, splits it on whitespace, and returns the last element (the last name). Very convenient!

You‘ll often see lambda functions used with Python‘s built-in functions like map, filter, and sort, but they can be used anywhere you need a small anonymous function.

4. Map and filter

Map and filter are two built-in functions that are incredibly useful in data science when you need to apply a function to a list (or other iterable).

Map takes a function and an iterable and returns a new iterable with the function applied to each element. For example:

numbers = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x**2, numbers))

Filter also takes a function and an iterable, but it returns a new iterable containing only elements for which the function returns True. To get only odd numbers:

numbers = [1, 2, 3, 4, 5]
odd_numbers = list(filter(lambda x: x % 2 != 0, numbers))

Together, map and filter allow you to write expressive data transformations in a functional style. And they‘re often more efficient than the equivalent for loops.

5. Generators and yield

Generators are functions that return an iterator object. They allow you to create iterators in a very concise way, using the yield keyword. For example, here‘s a generator that yields the Fibonacci sequence:

def fib():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

To use it:

for num in fib():
    if num > 1000:
        break
    print(num)

The big advantage of generators is that they allow you to work with sequences without generating and storing the entire sequence in memory. This is crucial in data science when you‘re dealing with huge datasets that would overwhelm your computer‘s memory if you tried to create a list of the entire dataset.

6. Decorators

Decorators are a way to modify or enhance functions without changing their definition. They‘re denoted with an @ symbol. For example, here‘s a decorator that logs the arguments and return value whenever the function is called:

def logging(func):
    def logged(*args, **kwargs):
        print(f"Called {func.__name__} with {args} and {kwargs}")
        result = func(*args, **kwargs)
        print(f"Returned {result}")
        return result

    return logged

@logging
def factorial(n):
    if n == 0:
        return 1
    else: 
        return n * factorial(n-1)

Now whenever factorial is called, it will print the arguments and return value:

factorial(5)

# Called factorial with (5,) and {}
# Called factorial with (4,) and {} 
# Called factorial with (3,) and {}
# Called factorial with (2,) and {}
# Called factorial with (1,) and {}
# Called factorial with (0,) and {}
# Returned 1
# Returned 1
# Returned 2
# Returned 6
# Returned 24
# Returned 120

Decorators are especially handy for adding functionality to existing functions in libraries or modules that you don‘t want to modify directly.

7. Context managers

Context managers define a runtime context that‘s entered before a block of code is executed and exited when the block is complete. They‘re commonly used to manage resources like file handles or database connections.

The most common way to write context managers is using the with statement. For example, here‘s how you‘d typically open a file:

file = open(‘example.txt‘)
data = file.read()
file.close()

But if an exception occurs between opening and closing the file, it won‘t be closed properly. The with statement ensures proper acquisition and release of resources:

with open(‘example.txt‘) as file:
    data = file.read()

No matter what happens inside the with block, the file will be closed when the block is exited. This is much cleaner and less error-prone.

You can also write your own context managers using the contextlib module. This is really useful when you need to set up and tear down resources in a way that‘s not already handled by Python‘s built-in context managers.

8. Magic methods

Magic methods (also called dunder methods) are special methods with double underscores before and after their name that you can define to give your objects certain behaviors. They‘re used to overload operators, customize attribute access, make objects callable, and more.

For example, by defining an __iter__ method, you can make your object iterable:

class Fibonacci:
    def __init__(self, max):
        self.a, self.b = 0, 1
        self.max = max

    def __iter__(self):
        return self

    def __next__(self):
        fib = self.a
        if fib > self.max:
            raise StopIteration
        self.a, self.b = self.b, self.a + self.b
        return fib

Now you can use your Fibonacci object in a for loop:

for num in Fibonacci(1000):
    print(num)

There are dozens of magic methods you can define, allowing you to customize your objects‘ behavior in all sorts of powerful ways.

9. Mixins

Mixins are a form of multiple inheritance where a subclass derives from two or more superclasses. They‘re used to compose behaviors from multiple parent classes.

For example, say you have a Serializable mixin that provides serialization and deserialization methods:

class Serializable:
    def serialize(self):
        return json.dumps(self.__dict__)

    def deserialize(self, json_data):
        self.__dict__ = json.loads(json_data)

You can then mix this into any other class that needs serialization capabilities:

class User(Serializable):
    def __init__(self, username, email):
        self.username = username
        self.email = email

user = User(‘johndoe‘, ‘[email protected]‘)
serialized_data = user.serialize()

Mixins allow you to write modular, reusable code and avoid the complexity of deep inheritance hierarchies. They‘re especially useful in data science projects where you often need to compose models, data loaders, visualizations, and other components in flexible ways.

10. Monkey patching

Monkey patching is the practice of modifying a module or class at runtime. It allows you to change code behavior without modifying the original source code, which is especially useful when working with third-party libraries that you can‘t or don‘t want to modify directly.

For example, say you want to add a new method to Python‘s built-in list class:

def average(self):
    return sum(self) / len(self)

list.average = average

Now you can call the average method on any list:

numbers = [1, 2, 3, 4, 5]
print(numbers.average())  # Output: 3.0

Monkey patching can be very powerful but it should be used judiciously as it can lead to unexpected behavior if not done carefully. It‘s most appropriate when you need to modify behavior globally and consistently across your codebase.

Conclusion

There you have it – 10 powerful Python tricks that will take your data science code to the next level!

We‘ve covered a lot of ground, from list creation with comprehensions, to anonymous functions with lambdas, functional-style operations with map and filter, memory-efficient sequence processing with generators, enhancing functions with decorators, safe resource handling with context managers, customizing object behavior with magic methods, composing behaviors with mixins, and runtime modifications with monkey patching.

Each of these tricks allows you to write cleaner, more concise, more efficient, and more expressive code. They‘ll help you get more done in less time and tackle more ambitious data science projects.

Of course, simply knowing these tricks is not enough – you need to put them into practice! I encourage you to revisit your old code and see where you could apply some of these techniques. And as you write new code, keep these tricks in mind and try to use them whenever appropriate.

If you want to go deeper into any of these topics, I‘ve compiled some additional resources below. Happy coding!

Additional resources:

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

You May Like to Read,

Similar Posts