Goal / Problem

The key goal that led to this solution is to have a single general implementation of a programming interface (we’ll refer to this core interface as the core) that can be the basis underlying various access points of that core interface. An access point for instance can be a Rest API exposed through URIs, or a custom command shell for a terminal, or a library interface to another language, etc (lets call these the channels for accessing the core). The interesting challenge to overcome is to expose these channels without redundant coding.

Naively, one could simply implement the core and channels then update the channels to reflect changes to the core as the core evolves. This approach however introduces a significant amount of coding complexity that negatively impacts maintainability. It would be elegant to have an approach for implementing a channel that would be implicitly congruent to the core. Basically, we’d like to have the channels auto-update as the core changes. Here are some maneuvers I took to achieve this with which I am very pleased. In this example we’ll focus on how this can be achieved with the implementation of a REST API interface to a core.

Assumptions

Though the techniques described here are general to python, some relevant context to the example described herein include

1. Django rest framework is used in this project to implement the Rest API interface

Technique

First lets take a look at the class that represents the key control interfaces that represent the core

core/master.py

"""
Main master handler for each user of Jaseci, serves as main interface between
between user and Jaseci
"""
from core.element import element
from core.graph.graph import graph
from core.graph.node import node
from core.actor.sentinel import sentinel
from core.actor.walker import walker
from core.utils.id_list import id_list
import json


class master(element):
    """Main class for master functions for user"""

    def __init__(self, email="Anonymous", *args, **kwargs):
        self.graph_ids = id_list(self)
        super().__init__(name=email, kind="Jaseci Master", *args, **kwargs)

    def api_create_graph(self):
        pass  # Omitted

    def api_create_sentinel(self, gph: graph):
        pass  # Omitted

    def api_delete_graph(self, gph: graph):
        pass  # Omitted

    def api_delete_sentinel(self, snt: sentinel):
        pass  # Omitted

    def api_get_jac_code(self, snt: sentinel):
        pass  # Omitted

    def api_set_jac_code(self, snt: sentinel, code: str, encoded: bool):
        pass  # Omitted

    def api_compile(self, snt: sentinel):
        pass  # Omitted

    def api_spawn_walker(self, snt: sentinel, name: str):
        pass  # Omitted

    def api_unspawn(self, wlk: walker):
        pass  # Omitted

    def api_prime_walker(self, wlk: walker, nd: node):
        pass  # Omitted

    def api_run_walker(self, wlk: walker):
        pass  # Omitted

    def destroy(self):
        pass  # Omitted

This class holds a key set of methods that are used as the primary interface to an underlying engine. From this core, a number of channels can be created including a cmd shell, Rest APIs, internal calls, etc. (Note that the preamble in the function names of the functions intended to be the api is api_. This is intentional for reasons that will be apparent below.)

But first, lets look at a typical example of how one might naively implement a channel for REST APIs (in Django style).

Naive Approach

This implementation represents an approach of using post request body parameters and piping them through to the core. The decode of the request before calling core is omitted for simplicity. Note that AbstractJacAPIView is a subclass of APIView from django-rest-framework.

jac_api/views.py

from rest_framework.response import Response
from .views import AbstractJacAPIView
from core.graph.graph import graph
from core.actor.sentinel import sentinel
import json


class call_create_graph(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_create_sentinel(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_delete_graph(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_delete_sentinel(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_get_jac_code(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_set_jac_code(AbstractJacAPIView):
    def post(self, request):
         pass  # Omitted


class call_compile(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_spawn_walker(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_unspawn(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_prime_walker(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted


class call_run_walker(AbstractJacAPIView):
    def post(self, request):
        pass  # Omitted

There are a number of issues here. Firstly, there must be custom code to process incoming request bodies so they can be parsed, decoded, and checked before hitting the actual core apis. Secondly, as the core interface changes, this REST code will have to be rewritten and maintained with this type of implementation. Pain in the buttocks!

A Better Way

Now let’s look at the better way. Our goal here is to automatically generate the implementation of the Rest API based on the implementation of our core. Lets take a look at how this can be done.

Firs, let’s re-imagine that views.py file to realize a self writing module that will generate classes for APIs in Django style (by subclassing AbstractJacAPIView which itself is a subclass of APIView). We put this in a separate module.

Generating REST APIs

jac_api/api.py

from .views import AbstractJacAPIView
from core.master import master
from core.utils.mem_hook import mem_hook
from inspect import signature
import types
import functools


def copy_func(f):
    g = types.FunctionType(f.__code__, f.__globals__, name=f.__name__,
                           argdefs=f.__defaults__,
                           closure=f.__closure__)
    g = functools.update_wrapper(g, f)
    g.__kwdefaults__ = f.__kwdefaults__
    return g


# Introspection on master interface to generate view class for master apis
for i in dir(master(h=mem_hook())):
    if (i.startswith('api_')):
        gen_cls = type(i,
                       (AbstractJacAPIView,),
                       {'api_sig': signature(getattr(master, i))})
        gen_cls.post = copy_func(gen_cls.post)
        gen_cls.post.__doc__ = getattr(master, i).__doc__
        globals()[i] = gen_cls

There are a number of interesting things going on in this module. Let’s break down the key parts.

# Introspection on master interface to generate view class for master apis
for i in dir(master(h=mem_hook())):
    if (i.startswith('api_')):

In these lines of code we inspect the member fields of the master class using the dir builtin and filter for only the functions that start with api_.

        gen_cls = type(i,
                       (AbstractJacAPIView,),
                       {'api_sig': signature(getattr(master, i))})

Here, we create an instance of a class object using the type built in. When called with three parameters in the format type(classname, superclasses, attributes_dict) a new type object is returned as described in python’s documentation on builtins.

With three arguments, return a new type object. This is essentially a dynamic form of the class statement. The name string is the class name and becomes the __name__[classname] attribute; the bases tuple itemizes the base classes and becomes the __bases__[superclasses] attribute; and the dict dictionary is the namespace containing definitions for class body and is copied to a standard dictionary to become the __dict__[attributes_dict] attribute. For example, the following two statements create identical type objects:

https://docs.python.org/3/library/functions.html#type

In the case of our api.py, we name the class the same name as the api function from core, we have it derive from AbstractJacAPIView, and then we use the attributes_dict parameter to ‘bake in’ the function’s signature as a member field api_sig so it can be referenced and analyzed as needed. In our case, as we show later in this article, api_sig is used to auto process a request body from a JSON post call to generate the internal function call that will hit our core interface. This code is held in the post function of AbstractJacAPIView and inherited into our generated class gen_cls.

However, for the sake of leveraging docstrings for documenting the API functionality, we’d like to pull the docstrings from our core interface and pipe it through to the Django style rest API implementation. We do that with the next two lines.

        gen_cls.post = copy_func(gen_cls.post)
        gen_cls.post.__doc__ = getattr(master, i).__doc__

As it turns out, we must do a full copy of the base function and rewrite it to the inheriting class as there is only one instance of the base post method in python’s memory. Simply assigning with gen_cls.post.__doc__ = getattr(master, i).__doc__ without the copy_func would result in all generated classes having the last assigned docstring value to AbstractJacAPIView’s base. Therefore, we copy the post function then assign that copy to the inheriting class to essentially override the base implementation with an exact copy of the base but new docstring from core.

        globals()[i] = gen_cls

Finally, we add the class to the api.py module buy setting the dict representing the fields of the module.

Viola! As we make changes to both the implementation of the core interfaces, the REST APIs stay congruent with both the interfaces themselves as well as the docstrings describing the interface.

Oh, but then there’s the task of reading the post requests to the REST API and generating calls to the core interface. This processing is automated in the base AbstractJacAPIView class.

Auto Patching Calls Through to the Core

jac_api/views.py

from rest_framework.views import APIView
from rest_framework.authentication import TokenAuthentication
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from core.utils.utils import logger
from core.element import element
import uuid


class AbstractJacAPIView(APIView):
    """
    The builder set of Jaseci APIs
    """
    authentication_classes = (TokenAuthentication,)
    permission_classes = (IsAuthenticated,)

    def post(self, request):
        """
        General post function that parses api signature to load parms
        SuperSmart Post - can read signatures of master and process
        bodies accordingly
        """
        self.proc_request(request)
        param_map = {}
        for i in self.api_sig.parameters.keys():
            if (i == 'self'):
                continue
            p_name = i
            p_type = self.api_sig.parameters[i].annotation
            param_map[i] = self.get_param(name=p_name, typ=p_type)
            if (param_map[i] is None):
                logger.error(self.res)
                return Response(self.res)
        return Response(getattr(self.master, type(self).__name__)(**param_map))

    def proc_request(self, request):
        """Parse request to field set"""
        self.cmd = request.data
        self.master = request.user.get_master()
        self.res = "Not valid interaction!"

    def get_param(self, name, typ):
        """Parse request to field set, req flags if required"""
        if (name in self.cmd.keys()):
            val = self.cmd[name]
            if (issubclass(typ, element)):
                val = self.master._h.get_obj(uuid.UUID(val))
                if (isinstance(val, typ)):
                    return val
                else:
                    self.res += f'{type(val)} is not {typ}'
                    return None
            else:
                return val
        else:
            self.res += f'\nInvalid API parameter set - {self.cmd}'
            logger.error(self.res)
            return None

The translation of request payload to function call is implemented in the post method. First we process the request to set a few field variables with the self.proc_request(request) method. We then cycle through the parameters of the core api using it’s signiture with for i in self.api_sig.parameters.keys():. We skip self of course (as it is not relevant to the API’s usage) and assume that each parameter in the core function signature must be passed through the REST API call. We collect the API’s call name and type (from type hint in core signature) and set to p_name and p_type. (remember self.api_sig is of type signature from Python’s signature module. self.api_sig.parameters[i].annotation is type corresponding to the type hint from the function signature).

For each parameter in the function signature we process using paparam_map[i] = self.get_param(name=p_name, typ=p_type). If this call returns None, there was an error with the JSON body of the post request in that it did not conform to the core api’s usage as defined by it’s signature. If self.get_param is successful we issue the call to the core api function with getattr(self.master, type(self).__name__)(**param_map) passing the parameters using the **param_map keyword dict. The return of the core call is then wrapped as a Response and returned from the post call.

The self.get_param function is a place for us to do any checking or converting of parameters coming from the JSON payload, in the case here we do a simple check to see whether the type expected for a param derives from an element base type and if so, do a lookup of the element to pass to the core api call. This is just one example of the type of processing that might be needed as we translate JSON to internal core api calls. Care should be taken to keep this as general and automated as possible.

Finally, we must expose these APIViews through URIs.

Generating URL/URIs for REST API

jac_api/urls.py

from django.urls import path
from . import api

app_name = 'jac_api'

urlpatterns = []

for i in dir(api):
    if(i.startswith('api_')):
        urlpatterns.append(
            path(f'jac/{i[4:]}', getattr(api, i).as_view(), name=f'{i[4:]}'))

Here is the realization of URL/URI path generation using Django style urls.py. Note that further introspection is happening here. We use dir on the api.py module and reference the classes we generated there with getattr(api, i).as_view().

Win

Voila! Now we only need to hack our core/master.py and our Rest APIs will stay congruent!

We have now autogenerated this

From our core implementation

"""
Main master handler for each user of Jaseci, serves as main interface between
between user and Jaseci
"""
from core.element import element
from core.graph.graph import graph
from core.graph.node import node
from core.actor.sentinel import sentinel
from core.actor.walker import walker
from core.utils.id_list import id_list
import json


class master(element):
    """Main class for master functions for user"""

    def __init__(self, email="Anonymous", *args, **kwargs):
        self.graph_ids = id_list(self)
        super().__init__(name=email, kind="Jaseci Master", *args, **kwargs)

    def api_create_graph(self):
        pass  # Omitted

    def api_create_sentinel(self, gph: graph):
        pass  # Omitted

    def api_delete_graph(self, gph: graph):
        pass  # Omitted

    def api_delete_sentinel(self, snt: sentinel):
        pass  # Omitted

    def api_get_jac_code(self, snt: sentinel):
        pass  # Omitted

    def api_set_jac_code(self, snt: sentinel, code: str, encoded: bool):
        pass  # Omitted

    def api_compile(self, snt: sentinel):
        pass  # Omitted

    def api_spawn_walker(self, snt: sentinel, name: str):
        pass  # Omitted

    def api_unspawn(self, wlk: walker):
        pass  # Omitted

    def api_prime_walker(self, wlk: walker, nd: node):
        pass  # Omitted

    def api_run_walker(self, wlk: walker):
        pass  # Omitted

    def destroy(self):
        pass  # Omitted