Beautiful is better than ugly.

Explicit is better than implicit.

Simple is better than complex.

Complex is better than complicated.

Complex is better than complicated.

Flat is better than nested.

Sparse is better than dense.

Readability counts.

Special cases aren't special enough to break the rules.

Special cases aren't special enough to break the rules.

Although practicality beats purity.

Although practicality beats purity.

Errors should never pass silently.

Errors should never pass silently.

Unless explicitly silenced.

In the face of ambiguity, refuse the temptation to guess.

In the face of ambiguity, refuse the temptation to guess.

There should be one-- and preferably only one --obvious way to do it.

There should be one-- and preferably only one --obvious way to do it.

Although that way may not be obvious at first unless you're Dutch.

Although that way may not be obvious at first unless you're Dutch.

Now is better than never.

Although never is often better than *right* now.

Although never is often better than *right* now.

If the implementation is hard to explain, it's a bad idea.

If the implementation is hard to explain, it's a bad idea.

If the implementation is easy to explain, it may be a good idea.

If the implementation is easy to explain, it may be a good idea.

Namespaces are one honking great idea -- let's do more of those!

Namespaces are one honking great idea -- let's do more of those!

'for Guido to fill in'ORit's an opportunity for people to provide their own addition.https://stackoverflow.com/questions/4504487/the-zen-of-python-distils-the-guiding-principles-for-python-into-20-aphorisms-bu#24814971

20th principle

Long time Pythoneer Tim Peters succinctly channels the BDFL’s guiding principles for Python’s design into 20 aphorisms, only 19 of which have been written down.https://peps.python.org/pep-0020/

import this

The Zen of Python, by Tim Peters

Dataclasses are an implementation of a code generator, similar to a Named Tuple. The idea of using a code generator is to save time and reduce wordiness.There are two views about the purpose of Dataclasses:<ol><li>It makes a mutable data holder, in the spirit of named tuples</li><li>It writes boiler-plate code for you, simplifying the process of writing the class</li></ol>These 2 world views are reflected in the name: “Dataclasses”Per the dataclasses PEP, they are roughly a “Mutable named tuple with defaults”They provide an elegant syntax for creating data holder objects

Dataclasses & What are they for

In the beginning there where:<ul><li>dicts for look-ups</li><li>tuples for structures</li><li>hand-written classes for more functionality</li></ul>Later Named Tuples where added as a code generator with the goal of adding names to the fields of a tuple. This was limited by design, hence its nameORMs appeared and pioneered using class attributes to specify rich data structuresIn Python 3.6, it became possible to supply <a href="https://deepstash.com/article/137403/an-introduction-to-python-type-hints">type annotations</a>

A History Lesson

<ul><li class="ql-align-justify">Dataclasses are by default mutable</li><li class="ql-align-justify">Named Tuples have methods which start with an underscore which can be confusing. Ex: obj._replace(f=value)</li><li class="ql-align-justify">Named Tuples have legacy code. Ex: _asdict() returns an OrderedDict</li><li class="ql-align-justify">Named Tuples can be unpacked</li><li class="ql-align-justify">Dataclasses are faster at retrieving data but take up more space</li><li class="ql-align-justify">Dataclasses use functions instead of methods. Ex: asdict() vs nt._asdict()</li><li class="ql-align-justify">Dataclasses are underlying stored as instance dicts</li></ul>

Comparison with Named Tuples

If you have an immutable mind get out!

<ul><li>Dunder methods like init, repr, eq, lt, le, etc.</li><li>Frozen nature by generating dunder methods for __setattr__ and __delattr__</li><li>Class variables for the provided attributes</li></ul>The code generation can be configured by providing keyword arguments to the @dataclass decoratorA complete list of methods and configurations can be found <a href="https://docs.python.org/3/library/dataclasses.html">here</a>

Generated Code Includes

In the image example, the generated code includes:<ul><li>typed attributes</li><li>hidden attributes from __repr__ response</li><li>custom __hash__ dunder with the selected fields</li><li>attribute with default factory value in __init__ constructor</li></ul>

A Complex Example of using Dataclasses

I started using Dataclasses and I think you should too

Speaker: Raymond Hettinger The PEP 557 dataclasses module is available in starting in Python 3.7. It will become an essential part of every Python programmer's toolkit. This talk shows what problem the module solves, explains its key design decisions, and provides practical examples of how to put it to work. Dataclasses are shown to be the next step in a progression of data aggregation tools: tuple, dict, simple class, bunch recipe, named tuples, records, attrs, and then dataclasses. Each builds upon the one that came before, adding expressiveness at the expense of complexity. Dataclasses are unique in that they let you selectively turn-on or turn-off its various capabilities and it lets the user choose the underlying data store (either instance dictionary, instance slots, or an inherited base class). Dataclasses and typing.NamedTuple both use variable annotations which were new in Python 3.6. Slides can be found at: https://speakerdeck.com/pycon2018 and https://github.com/PyCon/2018-slides

Backend

Dataclasses:  The code generator to end all code generators

This article is recommended to be read having some prerequisites. You should be familiar with HTTP Methods and some basics regarding Django.If that's not your case, please check the resources below before proceeding with the article:<ul><li><a href="https://www.restapitutorial.com/lessons/httpmethods.html">HTTP Methods</a></li><li><a href="https://docs.djangoproject.com/en/4.0/intro/tutorial01/">Django Tutorial (7 parts)</a></li></ul>

Introduction

Tastypie is a  framework for Django that helps programmers to develop RESTful APIs on top of their already existing projects.

What is Tastypie

As a general rule, each group of endpoints is seen as a resource. When I say a group of endpoints I mean a collection of HTTP methods on a specific route (e.g. GET, POST and DELETE on /api/entity).These resources have a lot of features, which include allowing you to access the endpoint with specific HTTP methods, permitting access only if you are authenticated, paginating the results etc.

What is a Tastypie Resource

In Tastypie, there are 3 types of endpoints:<ol><li>list</li><li>detail</li><li>custom</li></ol>List endpoints return a list of objects (as the name implies) and it looks like `host_address/api/entity`. Detail endpoints return a single instance of an object and it looks similar to the list endpoint, except for the fact that it expects a path param, the ID of the object, to identify the instance that will be returned (e.g. `/api/entity/[id]`).Custom endpoints are routes made by developers to solve any use-cases different from the default ones. They don't have a specific format, offering devs more freedom.

Types of endpoints

Each endpoint needs a function to manipulate and return the data. But these functions have a standard format<ul><li>They have as parameters `self` (being a class method inside the resource), `request` (which is the `Request` instance of the client's API call</li><li>They expect the returned value to be a HttpResponse. For best-practice and convenience, there is a method called `create_response` that takes care of that. All you need to pass is the request and the object that you want to be included</li></ul>

Endpoint function structure

Now let's focus on what's the steps for a request to be fulfilled on a Tastypie endpoint.When a request is received from a client, Django checks for any existing routes inside `urlpatterns`, the variable with all the available routes inside the project.If a pattern was found, Django will call the associated view (function or class) with the following parameters:<ol><li>an instance of HttpRequest</li><li>extra params got from the URL</li></ol> If the URL didn't match any pattern, then Django will take care of it via an appropriate error handling view.

Routing a request

Once Django redirects the request to a Tastypie resource, it checks for the available URLs and lets Tastypie do the rest.  Assuming the URL is`host-domain/api/v1/entity/` (which is a list view):If the view was matched, `Resource.wrap_view('dispatch_list')` is called. The wrap_view method inside the Resource takes care of basic error handling and serialisation for the errors. The 'dispatch_list' parameter means that Resource.dispatch_list will be called next (a thin wrapper around the dispatch method).

Handling a request inside Tastypie

The dispatch method makes sure the followings are fulfilled:<ul><li>the HTTP Method inside the request is allowed in the resource</li><li>the resource has a method that can handle the request (e.g. get_list, get_detail, get_custom_endpoint)</li><li>the user is authenticated</li><li>the user did not exceed their throttle</li></ul>

Resource.dispatch method

The response is generated using the specific class method of the resource. If it's a default endpoint, the methods look like [HTTP_method]_[view_type] (e.g. get_list, post_detail) and if it's a custom endpoint, the method's name is the defined route for the endpoint (e.g. add_more_entries)

Response generators methods naming

The following actions will be done when calling the get_list:<ol><li>All the available objects are fetched via `Resource.obj_get_list`</li><li>Objects are sorted via `Resource.apply_sorting` and paginated with the built-in Paginator</li><li>the data will be dehydrated, meaning Tastypie will keep only the allowed fields</li><li>`Resource.create_response` is called, which serializes the data in the proper format and returns a HttpResponse object with the data</li></ol>Observation: For POST/PUT, there will be an extra step, the hydrate cycle, that will take the request payload and convert it in raw data, for DB storage

Generating the response for `get_list`

Once the Tastypie returned the HttpResponse object, Django will send it to the client which made the request.As a conclusion, you can consider Tastypie endpoints some Django views which return processed data in standardized formats (or HTTP errors, based on the case).

My takes after reading the Tastypie documentation, regarding how a request is handled inside an endpoint

Tastypie documentation

<ul><li>Dynamic : pipelines are configuration as code, allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.</li><li>Extensible : Easily define your operators, executors and extend the library so that it fits the level of abstraction that suits your environment.</li><li>Elegant : pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful Jinja templating engine.</li><li>Scalable : has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.</li></ul>

What are the principles of Airflow?

"<a href="https://airflow.apache.org/docs/apache-airflow/stable/index.html">Airflow </a>is a platform to programmatically author, schedule and monitor workflows"

It is used for the scheduling and orchestration of data pipelines or workflows as directed acyclic graphs (DAGs) of tasks.

The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies.

The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

"Airflow is a platform to programmatically author, schedule and monitor workflows"

It is used for the scheduling and orchestration of data pipelines 

Apache Airflow in Data Engineering

is the process of repeating items in a self-similar way. In programming languages, if a program allows you to call a function inside the same function, then it is called a recursive call of the function.

Recursion

<a href="https://en.wikipedia.org/wiki/Recursion">Recursion </a> occurs when a thing is defined in terms of itself or of its type. Recursion is used in a variety of disciplines ranging from linguistics to logic. The most common application of recursion is in mathematics and computer science, where a function being defined is applied within its own definition. While this apparently defines an infinite number of instances (function values), it is often done in such a way that no infinite loop or infinite chain of references can occur.

Recursive Functions is Inception, where function can be assumed of dream,where the function can call itself until the base result is obtained.

The Zen of Python, by Tim Peters

Dataclasses: The code generator to end all code generators

Tastypie documentation

Apache Airflow in Data Engineering

Recursion

It's time to

READ

LIKE

A PRO!