How I Solved My First Circular Dependency Problem in Python

Googling.

I mean, DuckDuckGoing.

That's it that's the answer.

Just kidding, that's the declarative version of the answer. Let's spend a little bit of time on the imperative version 😎.

A bit of background first.

Cutesy_MVC

Recently, in my spare time, I've been building a relatively miniature MVC framework in Python on Replit. A course I have right now at university is my first serious exposure to Python, and I find the language to be quite charming as it seems to encompass some of my favorite features from other languages like C++, JavaScript, PHP, and Ruby. As such, I figured it would be fun to attempt to recreate core features of the API of a framework that is dear to my heart: Laravel.

Laravel is a large and mature framework in the PHP world, and a blast to work with. I thought it might be fun to try to recreate in a smaller way some of the important features from Laravel (namely the "Model" and "Controller" components; for the View layer, I have in mind something a bit different than Laravel's beloved Blade templating system).

Of course, being a solo project I'm literally coding from my phone in my spare time, this isn't going to be a "clone" of Laravel. It's going to be more like the obscure generic off-brand ripoff of Laravel from the dusty old grocery store across town that makes you worry about the risk of contracting botulism. But whatever monster I end up making, I'm sure I will cherish it like my own personal 🧟‍♂️ Frankenstein 💚.

My Progress So Far

The first piece of inspiration came from the textbook we are using in my Python class. It's really a great book.

20210316_175651.jpg

Might actually take them up on their offer to keep it at the end of the semester 🤓. Sometimes it's handy to have a quality reference printed on the kind of dead trees that exist outside of cyberspace.

It has a great chapter on SQLite that inspired me to surf the SQLite website. Well, let me tell you, this isn't my first rodeo involving SQLite or relational databases, but it is the first rodeo where I was inspired enough to jump on the back of the bull myself and try to create my own database query builder instead of living vicariously through other people's admittedly more professional attempts to ride the bull.

First, like Laravel, I would need a simplified migration system.

✅ Done.

Don't look at it, it's fragile and hideous 🤢, and I don't want to talk about it other than to say it's riddled with attempts at premature optimizations and YAGNI rule violations and laughable design.

But hey, it has the minimum viable set of working features to allow me to migrate SQLite Databases and start building/testing the more interesting features of a query builder.

¯\_(ツ)_/¯

Second, I need a query builder system that can both be used on its own by framework users and to power the primitive ORM system this article is about.

✅ Done.

This one isn't that bad actually. Not perfect, but not horrible. I basically took a subset of the features of Laravel's query builder features that I felt were important and/or interesting and implemented them in a way that let's be honest is probably inferior to Laravel.

That being said, I'm happy with how it works, and I'm able to use it to power idk like 90-95% of the queries the ORM generates.

Third, I "need" (lol) to build an Object Relational Mapper (ORM).

👆 I'm almost done with it.

Before I talk about the circular dependency problem I encountered and solved while designing it I mean lazily coding it on my phone by the seat of my pants, I'll tell you a little bit about it generally.

I wanted to define a base class called Model that users will extend to define their own derived models. The base class is to provide all the ORM functionality that the user models will utilize to more fluently and succinctly query the database.

Aha, fluent queries, that must be why Laravel's ORM is called Eloquent. I guess that means I should call my own cheesy little ORM something appropriate as well. Maybe Illiteracy ORM?

Anyway I basically used a lot of Laravel's naming conventions, but I'm only implementing a subset of Eloquent's features — and in a different way than Eloquent does as well I'm sure.

One of the core features of an ORM is allowing users to easily relate objects to one another. Each instance of an ORM model is intended to wrap a bunch of convenient functionality around a single record from a database, whose many records can relate to one another in a bunch of different ways.

Hence, as an example, if records in one database table have a one-to-many relationship with records in another table, you want any one model object representing a record from the first table to easily be able to generate and access the objects that represent the many records it is related to in the second table.

I wanted users to be able to define their relationships directly as attributes on their model classes. However, as in the following example, I quickly ran into a circular dependency issue:

# MamaHen.py
from .Egg import Egg

class MamaHen:
  laysMany = Egg


# Egg.py 
from .MamaHen import MamaHen

class Egg:
  hasOne = MamaHen 


# __main__.py
from .MamaHen import MamaHen 

print("This print statement isn't even executed before an error is thrown!")

The first thing that came to mind to solve this problem is monkeypatching.

Monkeypatching is a Fun Novelty, But This Probably Isn't the Proper Use Case We're Looking For

If you come to a language like Python from object-oriented programming in a language like C++, you may be tempted to attempt to solve a circular dependency problem by trying out the easy monkeypatching capability that languages like Python and Ruby offer.

Classes are defined at runtime in Python, and you can modify them on-the-fly based on runtime conditions, such as a condition encountered within a single instance object.

Not to belabor the point, but I want to be clear about the distinction I'm making. Any individual object that has been instantiated can be modified at runtime — of course, that's kind of the point of stateful objects. However, a class, or the "blueprint" such an object was based on, can also be modified at runtime. That is the essence of monkeypatching.

Any instance object can access and mutate its class's attributes through self.__class__. If one instance object mutates its class at runtime, all of its sibling instance objects, even if already instantiated, will subsequently be able to see that mutation. Truthfully, this is not that dissimilar from the prototypal inheritance of a language like JavaScript.

# MyClass.py 
class MyClass:
  # class attribute
  numInstances = 0
  # instance constructor
  def __init__(self):
    self.__class__.numInstances += 1
  # instance destructor
  def __del__(self):
    self.__class__.numInstances -= 1


# __main__.py 
from .MyClass import MyClass 

def numObjects():
  print(f'Number of instances of MyClass: {MyClass.numInstances}')

numObjects() # 0

print('Instantiating 3 MyClass objects...')

x = [MyClass() for i in range(0,3)]

numObjects() # 3

print('Deleting 1 of the MyClass objects...')

del x[2]

numObjects() # 2

print('Deleting the remaining two instances...')

del x[1]
del x[0]

numObjects() # 0

So you reason, if the program is encountering a circular dependency problem rooted in class definitions at runtime, then we mightn't we be able to solve this problem at runtime as well through some clever hacks?

After all, part of the circular dependency problem is that a class definition depends on another class being defined beforehand. So it's really an ordering problem. Can we just conditionally define classes based on which have been defined already?

Indeed we can. Here's an example of an orderly class definition procedure:

# classDefiner.py 
def defineClassesInOrder(order):
  defined = []
  orderIndex = 0
  while len(defined) < 3:
    if order[orderIndex] == 1:
      class ClassA:
        def __str__(self):
          return "I'm ClassA."
      defined.append(ClassA)
    elif order[orderIndex] == 2:
      class ClassB:
        def __str__(self):
          return "I'm ClassB."
      defined.append(ClassB)
    elif order[orderIndex] == 3:
      class ClassC:
        def __str__(self):
          return "I'm ClassC."
      defined.append(ClassC)
    orderIndex += 1
  return defined[0], defined[1], defined[2]


# __main__.py 
from .classDefiner import defineClassesInOrder

# let's pretend the order of numbers in the following tuple is defined from user input (change the order if you like)
order = (2, 3, 1)

One, Two, Three = defineClassesInOrder(order)

print(One()) # ClassB
print(Two()) # ClassC
print(Three()) # ClassA

Our circular dependency problem is a little bit different though. Two or more of the class definitions literally both require the other to be defined first, unlike in that example. This is a condition that does not seem possible to logically fulfill.

At this point, it becomes clearer why Laravel has users define relationships within class methods instead of simple attributes. That way, the classes can be defined and at runtime relationships are defined on the fly when methods are called. I mean, don't quote me on Laravel working that way under the hood because I've never reviewed the code, but that's my guess 😅.

Maybe I'm being petty, but I felt like there had to be a way of making it work without instance objects defining relationships on-the-fly. To me it seems that the relationships logically belong to the classes and not the objects, so there should be a way of explicitly defining the relationships at the time the classes are defined.

An Unique and Immutable Value Having One-To-One Mapping to a Class Definition is Explicit Enough

I had a moment where I realized something about the dictionaries within which I was defining relations. Like any dictionary in python, the keys have to be unique, immutable values. Of course, the values in dictionaries don't generally have to follow that restriction, but for my use case it made sense to consider that really I wanted the values stored in the model keys in my relationship dictionaries to be unique and immutable as well.

After all, no user of my crappy little framework is going to be defining multiple identical models with the same name. And for polymorphic relationships, the model a record is related to is calculated at runtime based on values pulled from the database. The type string pulled from the database for polymorphic relationships is used as the lookup key in a dictionary to find the appropriate model class to use for the relationship—

🤔🤔🤔

Hey, maybe that's the solution.

I figured out that it's explicit enough to define a registry of strings or really any unique and immutable values that have one-to-one mappings to the model classes that take part in relationships, as long as they don't cause new circular dependencies 😂. It's explicit enough to define relationships at class definition time in terms of these strings, and then, when it comes time to hydrate models based on relationship definitions, it's not a very costly computation to utilize a lookup function that searches that registry for the needed model class definition to use for instantiation.

Here's a simplified example:

# Base.py 
class Base:
  def getChildType(self):
    from ._index import registry
    return registry[self.__class__.hasMany]
  def getOwnerType(self):
    from ._index import registry
    return registry[self.__class__.belongsTo]


# Fembot.py 
from .Base import Base

class Fembot(Base):
  hasMany = 'Wile'


# Wile.py 
from .Base import Base

class Wile(Base):
  belongsTo = 'Fembot'


# _index.py 
from .Fembot import Fembot
from .Wile import Wile

registry = {
  'Fembot': Fembot,
  'Wile': Wile 
}


# __main__.py 
from .Fembot import Fembot 
from .Wile import Wile 

print('Fembots have many:')
print(Fembot().getChildType())
# <class 'Wile.Wile'>
print('A single Wile may belong to only one:')
print(Wile().getOwnerType())
# <class 'Fembot.Fembot'>

We have successfully avoided the problem of circular dependency, while still explicitly defining inherently circular relationships between classes.

( ͡° ͜ʖ ͡°)

( ͡° ͜ʖ ͡°)>⌐■-■

ヾ(⌐■_■)ノ

It's important to note that the extraction of the getOwnerType and getChildType methods to the Base class, while sensible, is not what solves the circular dependency issue. If that extraction itself were the solution, and we had used top level imports in that file, we would still have the circular dependency issue because we would have in effect just made the circle larger. What actually solves the problem is sticking the import statements inside of the method definition, because then the import happens at runtime whenever that method is executed.

Wait... that sounds an awful lot like how Laravel avoids circular dependency. Oof. So much for originality lol. Well anyway, I still get to define the relationships explicitly in the class definitions, and the overhead of retrieving the references to the class definitions from the registry when needed is pretty low.

All the code examples in this article can be found in this REPL.