Scailable.

Instant, Production Ready, AI

← to Scailable.net

Apples and Oranges: Benchmarking Scailable


In one of Susan Li's "Toward Data Science" blog posts, she shows how to train classifiers on a fruit classification task with Python's Scikit-Learn. In the current post, we will demonstrate how Scailable allows you to put Susan's models into production with just one additional line of code. But before we get to that, let's first fit a Decision Tree Classifier using Susan Li's original code:

                from sklearn.preprocessing import MinMaxScaler
                from sklearn.model_selection import train_test_split
                from sklearn.tree import DecisionTreeClassifier
                import pandas as pd

                fruits = pd.read_table('data/fruit_data_with_colors.txt')

                feature_names = ['mass', 'width', 'height', 'color_score']

                X = fruits[feature_names]
                y = fruits['fruit_label']

                X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
                scaler = MinMaxScaler()
                X_train = scaler.fit_transform(X_train)
                X_test = scaler.transform(X_test)

                estimator = DecisionTreeClassifier().fit(X_train, y_train)

                estimator.predict_proba([[1.02142857, 1.11764706, 0.8, 0.51351351]])[0]
            
The last line results in the following prediction (representing the probabilities of a piece of fruit being an apple ๐Ÿ, a tangerine ๐Ÿฅญ, an orange ๐ŸŠ, or a lemon ๐Ÿ‹):

            > [0. 0. 1. 0.]
            
In other words, the fitted model predicts a piece of fruit with these particular features to be an orange ๐ŸŠ! Which is great - but how do you now bring this model into production to classify some fruit in real life? The answer is simple: install our SclblPy package and use one line of Python to make your model available through our Scailable API:
sclblPy.upload(estimator)
This one line of code uploads your model to our toolchain server, which transpiles the model into WebAssembly and makes it available through our Scailable REST API. Just use the UUID returned by the sclblPy upload function to obtain inferences from your fitted model through Scailable's API:
            > UUID: c5dfvre45
            
More specifically, the Scailable API enables you to configure, monitor, compare and invoke your fitted models in several ways, for instance by posting a feature vector in JSON format with cURL:

            curl --location --request POST 'https://demo.sclbl.net:8443/run/c5dfvre45' \
                --header 'Content-Type: application/json' \
                --data-raw '{"input": [[1.02142857, 1.11764706, 0.8, 0.51351351]]}'

            
.. or, for instance, using JavaScript:

            var myHeaders = new Headers();
            myHeaders.append("Content-Type", "application/json");

            var raw = JSON.stringify({"input":[[1.02142857,1.11764706,0.8,0.51351351]]});

            var requestOptions = {
            method: 'POST',
            headers: myHeaders,
            body: raw,
            redirect: 'follow'
            };

            fetch("https://demo.sclbl.net:8443/run/c5dfvre45", requestOptions)
            .then(response => response.text())
            .then(result => console.log(result))
            .catch(error => console.log('error', error));

            
... both returning:
            > {"output": [0.000000,0.000000,1.000000,0.000000]}
            
... that is, the WebAssembly invocation results in the exact same prediction: an orange ๐ŸŠ! Another option would be to not send your data to the fitted model, but the fitted model to your data. Scailable offers you the choice to do either, or both.

To support the former, we provide a K3S based Scailable Cloud ComputeNode infrastructure that makes use of TaskServers to scale your fitted model to up to tens of thousands of inferences per second through our REST API.

For the latter we offer lightweight run times, that allow you to interface with your transpiled fitted WebAssembly model through most any programming language and on most any device - from tiny IoT devices, to smartphones, to your client's browsers.

Each of these approaches has its advantages and disadvantages. Which plays to another strength of the Scailable platform: Scailable offers you the option to dynamically decide whether to run a model locally or remotely. Use preset rules or machine learning algorithms to determine if you want your fitted model to run on a client's mobile phone, on your servers, on one of the major cloud providers platforms (Google Cloud, Amazon AWS, Microsoft Azure), or, for example, as pictured above, on a smart doorbell type IoT device (specifically, our Python Sklearn fitted fruit model running on an ESP32 MCU board):

To get a better feel of local versus remote inference, just select one of the live versions of our fitted WebAssembly Decision Tree Fruit Classifier by selecting one of the buttons below. The first one downloads the fitted WebAssembly model to your browser and runs it locally in a JavaScript based ComputeNode - the second invokes the same model, but runs it remotely on a ComputeNode in our Scailable Cloud:


...

Which, on my Lenvovo T580 laptop over Wifi on a cable router in the Netherlands, gives the following results:

Network traffic Network time Compile time Run time
In-browser 46KB wasm
8KB javascript
± 15 ms ± 15 ms
Cloud 295 Bytes ± 35 ms 0 ms ± 65 ยตs
Of particular interest here is the fast computation of the prediction function when running on a Scailable Cloud ComputeNode. In contrast to running WebAssembly on the in-browser ComputeNode, running it on one of our Cloud ComputeNodes allow us to precompile the generated WebAssembly code. As can be deduced from the 65 ยตs computation time, this makes the WebAssembly model run at near-native C code speed.

To give you a better picture of how this allows us to scale with ease, we used loader.io to benchmark our Fruit Decision Tree model by invoking a thousand function calls a second, limiting our ComputeNodes to the confines of just one mid-sized virtual server (Ubuntu Linux 18.04, 4cpu, 16GB ram):

Clearly, our ComputeNodes have no problem whatsoever with this kind of load (the short dip represents a mostly resolved minor Golang related Garbage Collection issue).

We are nevertheless already working on the next iteration of our Cloud Infrastructure - in which we are experimenting with a unikernel based approach. Preliminary results indicate this approach will make our framework even lighter, faster, and more flexible - so be sure to stay tuned for more Scailable awesomeness!


Addendum: Some more Sklearn examples


Susan Li's logistic regression classifier


Of course, Scailable does not only support Decision Tree Classifiers, but all of the most often used Sklearn types of models. For instance, Susan's logistic regression from the original post:

                from sklearn.preprocessing import MinMaxScaler
                from sklearn.model_selection import train_test_split
                from sklearn.tree import DecisionTreeClassifier
                import pandas as pd

                fruits = pd.read_table('data/fruit_data_with_colors.txt')

                feature_names = ['mass', 'width', 'height', 'color_score']

                X = fruits[feature_names]
                y = fruits['fruit_label']

                X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
                scaler = MinMaxScaler()
                X_train = scaler.fit_transform(X_train)
                X_test = scaler.transform(X_test)

                estimator = LogisticRegression().fit(X_train, y_train)

                estimator.decision_function([[1.02142857, 1.11764706, 0.8, 0.51351351]])[0]
            
... leads to the following result:
        >  [1.456221, -2.533518, 1.609453, -0.532156],
    
Let's upload the model to Scailable again:
            sclblPy.upload(estimator)

            > UUID: 10wwcjmwf
    
So, let's now run the model on our Scailable cloud!
        curl --location --request POST 'https://demo.sclbl.net:8443/run/10wwcjmwf' \
        --header 'Content-Type: application/json' \
        --data-raw '{"input": [[1.02142857, 1.11764706, 0.8, 0.51351351]]}'
    
This results in:
    {
       "output": [1.456221,-2.533518, 1.609453, -0.532156 ],
       "fntime": "65.092ยตs"
    }
    
The exact same answer we got in Python! The logistic regression classifier computation time again equals native speeds. And the size of the WebAssembly file still measures about 46KB as well. Which is not surprising, as it mostly consists of C standard library (or "libc") functions we use in our inference function.

A larger linear regression model


So lets try another type of model, for instance a slightly more complex linear regression model, and see how that fares:

                from sklearn.datasets import load_boston
                from sklearn import linear_model

                boston = load_boston()
                X, y = boston.data, boston.target

                estimator = linear_model.LinearRegression()
                estimator.fit(X, y)

                estimator.predict ([[2.7310e-02, 0.0000e+00, 7.0700e+00, 0.0000e+00,
                                     4.6900e-01, 6.4210e+00, 7.8900e+01, 4.9671e+00,
                                     2.0000e+00, 2.4200e+02, 1.7800e+01, 3.9690e+02,
                                     9.1400e+00]])[0]
            
... another model, another data set, so, also another prediction:
        >  25.025562379053138
    
Send the model to Scailable again:
            sclblPy.upload(estimator)

            > UUID: kq2fbedhe
    
So, let's now run the model on our Scailable cloud!
curl --location --request POST 'http://localhost:8090/run/kq2fbedhe' \
--header 'Content-Type: application/json' \
--data-raw '{"input": [[2.7310e-02, 0.0000e+00, 7.0700e+00, 0.0000e+00, 4.6900e-01, \
        6.4210e+00, 7.8900e+01, 4.9671e+00, 2.0000e+00, 2.4200e+02, 1.7800e+01,\
        3.9690e+02,  9.1400e+00]]}'
    
This results in:
    {
    "output": [25.025562],
        "fntime": "17.8ยตs"
    }
    
I guess it is not a big surprise anymore: again, the same answer we got in Python! Very fast computation. And file size remains about the same.

One more time: an SVM regression model


Just for fun, one more try! This time, let's fit an SVM model, on the same Boston data set as we used in our linear regression example.

                from sklearn.datasets import load_boston
                from sklearn import svm

                boston = load_boston()
                X, y = boston.data, boston.target

                estimator = svm.SVR(gamma=0.001, C=100.)
                estimator.fit(X, y)

                estimator.predict ([[2.7310e-02, 0.0000e+00, 7.0700e+00, 0.0000e+00,
                                     4.6900e-01, 6.4210e+00, 7.8900e+01, 4.9671e+00,
                                     2.0000e+00, 2.4200e+02, 1.7800e+01, 3.9690e+02,
                                     9.1400e+00]])[0]
            
... which, as expected, predicts a slightly different y as compared to the previous linear regression:
        >  21.500008271560503
    
Let's upload to Scailable once again:
            sclblPy.upload(estimator)

            > UUID: w4k286uh
    
Running the model in our Scailable cloud:
curl --location --request POST 'http://localhost:8090/run/kq2fbedhe' \
--header 'Content-Type: application/json' \
--data-raw '{"input": [[2.7310e-02, 0.0000e+00, 7.0700e+00, 0.0000e+00, 4.6900e-01, \
        6.4210e+00, 7.8900e+01, 4.9671e+00, 2.0000e+00, 2.4200e+02, 1.7800e+01,\
        3.9690e+02,  9.1400e+00]]}'
    
This results in:
        {
        "output": [21.500008],
        "fntime": "47.4ยตs"
        }
    
Same answer. About the same speed. The file size is a bit larger: 122KB, or 47KB gzipped. About the size of an average optimized online image file - still not bad at all!

Actual edge computing using ridge classification


Finally, to finish things off, a model fitted using a ridge classifier with built-in cross-validation. This time, we'll make use of the classic "breast cancer" binary classification dataset with 30 features:

                from sklearn.datasets import load_breast_cancer
                from sklearn.linear_model import RidgeClassifierCV

                data = load_breast_cancer()

                X, y = data.data, data.target
                # outcome value: malignant (0) vs. benign (1)

                estimator = RidgeClassifierCV().fit(X, y)

                print(estimator.decision_function([[2.057e+01, 1.777e+01, 1.329e+02, 1.326e+03, 8.474e-02, 7.864e-02,
                                                    8.690e-02, 7.017e-02, 1.812e-01, 5.667e-02, 5.435e-01, 7.339e-01,
                                                    3.398e+00, 7.408e+01, 5.225e-03, 1.308e-02, 1.860e-02, 1.340e-02,
                                                    1.389e-02, 3.532e-03, 2.499e+01, 2.341e+01, 1.588e+02, 1.956e+03,
                                                    1.238e-01, 1.866e-01, 2.416e-01, 1.860e-01, 2.750e-01, 8.902e-02]]))
            
        >  -0.587891
    
Let's upload to Scailable once again:
            sclblPy.upload(estimator)

            > UUID: zdtf6e5v6
    
Running the model in our Scailable cloud:
curl --location --request POST 'https://demo.sclbl.net:8443/run/zdtf6e5v6' \
--header 'Content-Type: application/json' \
--data-raw '{"input": [[2.057e+01, 1.777e+01, 1.329e+02, 1.326e+03, \
        8.474e-02, 7.864e-02, 8.690e-02, 7.017e-02, 1.812e-01, \
        5.667e-02, 5.435e-01, 7.339e-01,3.398e+00, 7.408e+01, \
        5.225e-03, 1.308e-02, 1.860e-02, 1.340e-02,1.389e-02, \
        3.532e-03, 2.499e+01, 2.341e+01, 1.588e+02, 1.956e+03, \
        1.238e-01, 1.866e-01, 2.416e-01, 1.860e-01, 2.750e-01, \
        8.902e-02]]}'
    
This results in:
    {
    "output": [
        -0.587891
    ],
    "fntime": "73.305ยตs"
    }
    
Again: small WebAssembly file size (46kb without gzip compression) and fast, c-like speeds! ๐Ÿš€ Computing on the edge, at native speeds, no hassle: that's Scailable ๐Ÿ˜Š๏ธ




Created in the Netherlands with ♥. For questions, contact us at go [at] scailable [dot] net.