Scikit-Learn (Part 19)
Neil Haddley • October 13, 2026
Training and deploying models at scale: serving a TensorFlow model with Docker over REST and gRPC, updating it live, shrinking it for a phone, and spreading training over several devices
In Part 18, an agent learned by trial and error. The last chapter of the Hands-On Machine Learning book is about what happens once a model has been trained: how to put it where other people and programs can use it, and how to train bigger models by spreading the work over several machines. Most of it is about Google's cloud and NVIDIA graphics processors, which I did not use, so I ran the parts that I could on my Mac and say below what I could not.
The example is small and made up for the purpose: the handwritten-digit classifier from earlier, saved and served by a program that other programs can ask. I started the server in Docker and sent it three test images over the network, and it answered with the right digits, 7, 2, and 1. I then saved an improved version of the model next to the first, and a minute later the same request was answered by the new one, without my restarting anything.
Key Points
The lesson: a saved model is a folder that describes its own inputs and outputs, so a server can use it without any of my code. A new version of a model is just a new numbered folder, and the server notices it and switches over by itself. A model can also be made about three and a half times smaller for a phone, by storing its numbers with less precision: mine went from 171,748 bytes to 47,928.
What I learned from it:
- A program that can send text to a web address can use the model. A second, faster way needs a library on the client's side, and gave identical answers.
- Spreading training over several devices changes where the work is done, not what is learned. On my Mac, which had no graphics processor that TensorFlow could use, the code ran as usual, on one device.
- I ran the parts that work on one Mac, and did not run the cloud, the cluster, or the NVIDIA parts.
The chapter also covers running a model in a web page, managing graphics processor memory, and tuning settings in the cloud, which I read about and did not run.
Quick Example
The notebook trains a small MNIST digit classifier, saves it as version 0001 in a folder named my_mnist_model, and serves it with the official TensorFlow Serving program. I ran the server with Docker, as the notebook describes, and mounted the model folder into it:
PYTHON
1subprocess.run(["docker", "run", "-d", "--rm", "--name", "tfserving", 2 "-p", "8500:8500", "-p", "8501:8501", 3 "-v", os.path.abspath("my_mnist_model") + ":/models/my_mnist_model", 4 "-e", "MODEL_NAME=my_mnist_model", "tensorflow/serving:2.17.1"])
Then I sent the server three test images as text, in a format called JSON (JavaScript Object Notation), to a web address, which is called a REST request, and it answered with the chance of each digit:
PYTHON
1request_json = json.dumps({"signature_name": "serving_default", 2 "instances": X_new.tolist()}) 3response = requests.post("http://localhost:8501/v1/models/my_mnist_model:predict", 4 data=request_json).json()
The server said 7, 2, and 1, with chances of 0.99, 0.99, and 0.98, which are the right digits. The same request over gRPC, a faster route that sends the numbers in a compact binary form and needs a library on the client's side, gave identical numbers. Here is everything I measured:
| Step | Result |
|---|---|
| Version 0001 (one layer of 100 neurons, 10 epochs) | 95.74% of the validation images right |
| Answers over REST and over gRPC for the first three test images | 7, 2, 1, identical |
| Version 0002 (two layers of 50 neurons, 10 epochs) | 95.92% of the validation images right |
| The same request a minute after saving version 0002 | the second digit's chance moved from 0.99 to 1.00: the server was now using version 0002 |
| The saved model converted for TensorFlow Lite | 171,748 bytes |
| The same model, converted with the default optimizations | 47,928 bytes |
Nothing was restarted when version 0002 appeared. I saved the new version into the folder, waited a minute, and sent the same request, and the numbers had changed.
What training means here
This chapter changes nothing about how a model learns. A model is still nudged by slopes on batches of examples. What changes is where the work is done. The book describes two ways to split it. In data parallelism, each device holds a copy of the model, each takes a share of the batch, and the slopes are averaged before every copy makes the same nudge. In model parallelism, one big model is cut into pieces, one per device.MirroredStrategy is the first kind. On a machine with several graphics processors (GPUs) it would copy the model to each one. My Mac's graphics processor is an Apple one, which the TensorFlow in this notebook does not use, so the notebook printed a warning that no GPU was detected, and MirroredStrategy used one device, the processor. The weights became MirroredVariable objects, special variables that keep copies in step, and the training ran as usual: 10 epochs of about 70 seconds each, ending at 96.35% training accuracy and 98.30% validation accuracy. It did not run faster, because there was nothing to spread the work over, but it is the same code that would.
My MiniGPT model has 826,433 numbers, which is small enough that none of this is needed: one machine trains it. The same ideas apply to the models behind large language models, which are split over thousands of devices, but I have not done that.
Running the notebook in Jupyter
The chapter is in the notebook that goes with the book, 19_training_and_deploying_at_scale.ipynb. I ran the parts I could on my own Mac: exporting the model, serving it with Docker, the REST and gRPC requests, the new version, the TensorFlow Lite conversion, the device checks, and MirroredStrategy. That took under half an hour. I started TensorFlow Serving with the Docker command that the notebook describes, instead of installing the Linux program that it uses by default. I did not run the sections on Vertex AI, which need a Google Cloud account; or the training on a TensorFlow cluster, the distributed hyperparameter tuning, or AutoML. TensorFlow in my setup did not use my Mac's graphics processor, so what the notebook says about NVIDIA GPUs, I know only from reading it.
A saved model describes itself: saved_model_cli lists its signature, what goes in and what comes out; below it, the Docker command that starts TensorFlow Serving, whose log says it is listening for REST requests on port 8501
A REST request to the server: three rows of ten chances, with the highest for the digits 7, 2, and 1
The same request over gRPC: the answer is the same, though it arrives as a tensor
No GPU here: asking for a GPU that does not exist falls back to the processor, and with soft placement switched off it raises an error instead
MirroredStrategy on one processor: 10 epochs, ending at 96.35% training accuracy and 98.30% validation accuracy
Next, Scikit-Learn (Part 20) looks back over the whole series: how every model learned, and what I would still use.