# Where should your AI run? An interactive exploration of edge, cloud, and everything in between, where you build an architecture, change the constraints, and see what breaks. ## Introducing the simulation Let’s give an ordinary doorway light a small job: notice when someone walks up and turn on before they reach the door. Even though this could be solved with a PIR sensor, a camera captures the scene, a model looks for a person, and the prediction determines what happens next. On paper, that is _three boxes_ and looks like a straightforward system, the architecture is only `Camera → Model → Result`. Then the result arrives late, or the model needs more memory than the device has, or connection drops. _Someone rightfully asks why the footage of their front door needs to leave the building_. Maybe, a shadow is mistaken for a person? Each of those problems changes **where** the work should happen. That is what we’re going to build upon: one small system that accumulates requirements until the model is only one of its moving parts. ### Cloud, edge, and hybrid at a glance | | Cloud | Edge | Hybrid | | ---------------- | -------------------------------- | ----------------------------- | ------------------------------------ | | **Response** | Compute + network delay | Local compute delay | Urgent work local; other work remote | | **Connectivity** | Required for inference | Can infer offline | Local essentials survive outages | | **Data** | Approved inputs leave the device | Data can stay on the device | Selected payloads cross the boundary | | **Compute** | Remote model capacity | Local memory and power limits | Split by workload | | **Operations** | Centralized model updates | Deploy updates to devices | Coordinate both halves | For ease of understanding, this doorway in question is deliberately ordinary. You can tell whether the light is useful without knowing anything about neural networks. That leaves us room to concentrate on the architecture alone. And as mentioned above, a motion sensor would ideally be enough for an actual light; we’re using a camera here because it gives us a manageable way to explore inference and its dependencies. Now for the fun part: as you read, _change the controls_. Your simulation choices carry forward, and the architecture diagram follows along. After each tool, save a constraint for **your own workload**. Those answers build your recommendation at the end; they are separate from the doorway simulation. On a small screen, use `Show path` to unfold it. `Reset workload` puts the original camera setup back without sending you all the way back here. > The predictions are scripted, the timings are illustrative and for the time being, we are not sending actual inference requests. We’ll only make the consequences of your choices visible here, then look at how a real model could take over. ## Start with the smallest useful system Before we decide where anything runs, we need to know what work we are asking it to do. The **input** is one frame from the doorway camera: a still image captured at a particular moment. The **model** examines that frame for evidence of a person. The **output** is a prediction and a score. For this experiment, we boil that down to `Person` or `Empty`. We only need to detect presence, not identify anyone. Try the three frames below. A person standing in clear view is the easy case. An empty doorway is useful too: the system needs to know when to leave the light alone. The shadow is the ambiguous case. It looks enough like a person for the fictional model to say `Person` with a score of `54%`. Changing the frame changes what the model says. That is the entire job of this first experiment. We have not yet decided whether the prediction is fast enough, trustworthy enough, or even available when somebody reaches the door. > Instead of comparing abstract “edge” and “cloud” boxes, we can ask whether the person at this doorway gets a light, how long they wait, and what happens when the system is unsure. For our first attempt, the camera sends its frame to a model in the cloud and waits for a reply. The action can wait until we have examined that reply. ## When the model waits on the network Our small remote model takes `20 milliseconds` to process a frame. The network round trip adds `240 milliseconds`. The result arrives after `260 milliseconds`, with most of that time spent getting to and from the model. Raise the network latency and watch the cloud path get slower. Then move the model onto the device beside the camera. That local processor takes `34 milliseconds` for the same small workload in our simulation, because it does not need a reply from the internet. That looks like a convincing argument for local inference. Now set the network latency to zero. The cloud processor finishes first: `20 milliseconds` against the device’s `34 milliseconds`. Moving the model removed a network dependency. It did not give the little computer a faster processor. Whether the move helps depends on how much waiting it removes and how much local work it adds. For our light, the number that matters is the time between a person appearing and a useful decision arriving. They do not experience “inference latency” separately from “network latency.” They experience the total delay before the light comes on. A real response budget also includes capture, preprocessing, transfer, and queueing. Measure the slow responses as well as the average; _a light that is prompt nine times out of ten still gets judged on the tenth_. Here we isolate inference and network time so you can see their relationship. > With the original small model, moving it onto the device looks promising. The next question is whether it still helps with a larger workload. ## The model has to fit before it can run Suppose we want to try a more demanding model. Perhaps it handles difficult frames better; we would have to test that separately. For now, the thing we know is that it asks _more_ of the device. Increase the weight size from `5 MB` toward `8 GB`. Then change the device’s `memory budget`. These are separate controls: the model’s requirements change independently of the device’s capacity. Two different limits appear here. The model can fit and still take too long to run. Or it can need more memory than the device has, in which case our local path produces **no result at all**. The download size only accounts for the model weights. Running it also takes memory for inputs, intermediate values, and the inference runtime. Our simulation adds `50%` to the weight size and reserves another `128 MB` as an assumption. Speaking of that, the rising inference time is an assumption too. File size alone cannot predict speed across different models. Architecture, precision, kernels, and accelerators matter. Think of this slider as a fictional family of increasingly demanding models on the same processor. The memory selector changes capacity only. Now try the `16 GB` budget. The large model now fits, but its local response time has not improved. We solved “_can it load?_” but still need to answer “_will it finish in time?_” Power draw and sustained heat would be on the list for a physical device as well. > A small local model might be enough. A larger remote one might earn its network delay. So far, that comparison assumes the network is available. Let’s remove that assumption. ## When the connection disappears Leave your model and memory settings as they are. Disconnect the network. The cloud path _stops_. The local path keeps working if the model fits. If you left the device with an oversized model, both paths can fail at once. The controls remember your choices; the capacity and connectivity constraints now apply together. For the doorway, we can now make a more precise requirement: _the essential light decision must survive an outage_. A summary of the day’s activity could wait for connectivity. Those jobs happen around the same camera, but they do not have to share a deadline or a processor. That gives us a reason to split the work. Keep enough capability locally to handle the essential task, and send less urgent work elsewhere when the connection is available. The local model must actually handle the essential task for this split to provide resilience. > If a frame defeats the local path during an outage, we still need a defined response. Hold the decision, use an appropriate default, or ask for help. We’ll give that uncertainty its own route shortly. Reconnect whenever you’re ready. With the connection restored, we can examine what crosses it. ## What needs to leave the device? Our first cloud path sends raw frames. That gives the remote model its input, but it also sends footage of the doorway across the network. Could we do some of that processing before the data leaves? We can move a different piece of work onto the device: an _encoder_. It turns each frame into a smaller feature representation that a compatible remote model can use. In our example, the raw stream is `24 MB per second`; the feature stream is `18 KB per second`. Move the encoder and watch the payload crossing the boundary change. The diagram shows what a cloud request would carry. **Current outbound** also respects the architecture you have built so far. A fully local decision sends nothing, neither does an offline connection, so check those if yours says `0`. If we later request cloud review, that request will use the encoder placement you choose here. We now have some processing near the camera and some in the cloud. That is a hybrid architecture, with an actual job assigned to each side of the boundary. Keeping the raw frames local is useful, but a compact representation can _still expose private information_. Features can still carry sensitive information. You need to understand what they reveal, who receives them, and how long they are retained. Fewer bytes answer the bandwidth question; they do not settle every question about the data. The two halves also have to agree on the representation. Our fictional remote model supports both raw frames and these features. A real remote model must be designed to accept the local encoder’s output. For a physical system, measure the stream at the actual frame rate and resolution. Draw the boundary and write down the payload. Naming the payload makes it possible to review the bandwidth and privacy implications of that boundary. > We have dealt with getting a result, keeping it available, and moving its inputs. Now we need to decide when a result should trigger the light. The ambiguous shadow from the first experiment gives us a useful test case. ## When should a prediction become an action? Until now, we have looked at predictions. Here we connect them to behavior: an accepted `Person` prediction turns the light on; an accepted `Empty` prediction keeps it off. Select the ambiguous frame and set the confidence requirement above its `54%` score. Watch the decision leave the action path. Below the threshold, the action is held. If the model is local and the network is connected, the system requests cloud review. If that route is unavailable, it holds for review without sending anything. In this experiment, requesting review never silently becomes permission to act. Now lower the threshold to `54%`. The same shadow is accepted as a person and the light turns on. We did not change the model or the frame. We changed the policy around the prediction. For a light, an unnecessary activation may be an acceptable inconvenience. _A missed person might be more annoying_. The threshold should reflect those costs, using representative examples and measured errors. A more consequential action deserves its own policy, even if it starts with exactly the same model score. Also, `80%` on this slider is not an `80%` correctness guarantee. Model scores need calibration before you interpret them as probabilities. Here the score simply gives us a visible way to route an uncertain result. > Try disconnecting the network again, then come back to the shadow. This time the outage changes the review path as well as the inference path. That is the system we have been building: earlier decisions still matter when the next requirement arrives. ## The architecture around the model Look at the architecture. The camera is still there. So is the model. Around them, we have accumulated a response budget, a memory limit, outage behavior, a data boundary, and a policy that decides whether a prediction gets to do anything. Move the model one last time. This time, judge the move against all of those choices. Does the path still do what the person at the door needs? Even though the model was the first box we drew, it no longer explains the system by itself. A cloud design can make sense when waiting is acceptable, the connection is dependable, and the _data may leave_. A local design can keep an essential decision available through _an outage_. **A hybrid design can divide the responsibilities, as long as its local path really can keep the promise written next to it**. The light has become a useful little test of each claim. “Works offline” means a fitting local model can still make the necessary decision. “Keeps raw frames local” means the outbound payload supports that statement. “Handles uncertainty” means the shadow has somewhere to go besides directly into an action. Those are behaviors you can test, and they give you a concrete basis for choosing where the work belongs. ## Giving the camera a real model The next useful step is to replace our scripted predictions with a real detector. As a shameless plug: Qualcomm AI Hub’s [YOLOv8-Detection](https://aihub.qualcomm.com/models/yolov8_det) is a good fit for this doorway; it takes an image, finds objects, and returns classes, locations, and confidence scores. Its model card describes the `YOLOv8-N` variant, a `640 × 640` input, and export paths for supported Qualcomm hardware, which I have been experimenting on top of leading up to this thought. Instead of choosing between a made-up `96%` `Person` score and a made-up `54%` `Empty` score, we could show actual detections from a small set of doorway images. We could compare the clear frame and the ambiguous frame, measure how long each takes, inspect the memory required on a named device, and carry those observations into the latency, offline, and confidence-policy experiments. AI Hub also gives us a path beyond a laptop demo: compile or export the detector for a target Qualcomm device, profile it there, and compare a model that is merely runnable with one that meets the system’s response budget. The model card lists smaller quantized variants, which would make the memory experiment concrete too. ## So where should yours run? Your constraints now have a destination. Use the recommendation below as a starting architecture, then test the promises it makes on your hardware. Review any unanswered questions to sharpen the result.
## Ending notes The doorway started with a camera and a model. What determined the architecture was the promise around them: how soon the light must respond, what must keep working offline, what data may leave, and what happens when the prediction is uncertain. Your workload will have different promises, but the same questions decide where its work belongs. Keep essential decisions local when they must survive a lost connection or meet a deadline the network cannot support. Use the cloud when approved data can leave and remote capacity earns its delay. Split the work when a tested local path can cover the essentials and the rest can wait. Treat that placement as a claim to test. Measure it on the hardware, disconnect the network, and try the ambiguous input. If no path meets the constraints, change the model, hardware, or requirement. The architecture is ready when the whole system can keep its promise.