A computer-use agent reads a screenshot or structured UI representation, chooses an action such as click or type, observes the changed interface, and repeats. It is useful when a system has no suitable API, when visual state matters, or when work crosses legacy applications.
GUI control is broad and fragile. Coordinates can drift, pages can contain prompt injection, sessions may expose private data, and one mistaken click can submit a form or accept a contract. The runtime must treat every action as a proposed tool call under policy.
Prefer a narrow API or purpose-built tool whenever it can express the task. Computer use is the fallback for interfaces that require human-style interaction, not the default integration for every website.
An API is usually faster, cheaper, easier to validate, and less sensitive to visual redesign. Use computer control when no adequate API exists, when the task depends on rendered state, or when a human workflow must be automated across several applications.
Split mixed workflows. A browser agent may navigate a legacy portal, while typed services handle calculations, permission checks, file storage, and final writes. The fact that the agent can click a button does not mean clicking should replace a safer backend operation.
A screenshot is a moment in time. The page may scroll, a modal may appear, an animation may move the target, or a delayed network response may replace the content. Capture a fresh observation before consequential actions and after every state-changing action.
Prefer semantic targets from the accessibility tree or DOM when the environment exposes them. If coordinates are required, combine them with visible labels, region bounds, and expected nearby text. Never reuse a coordinate after navigation without re-observing.
The agent should state what it expects to change. Verification can check the URL, page title, confirmation text, changed record status, downloaded artifact, or backend state. A click is not evidence of success.
Web pages, emails, documents, ads, and images may contain instructions aimed at the model. That content is untrusted data even when it appears inside an application the user normally trusts. It must not redefine the task, reveal secrets, or grant permission.
Keep the user goal and system policy in a trusted channel. Restrict navigation to allowed domains, block secret entry into unapproved origins, and stop when the page asks for credentials, downloads an unexpected file, or presents instructions unrelated to the task.
A classifier or prompt warning is only one layer. Isolation, least privilege, domain policy, action confirmation, and output verification remain necessary because detection can miss adversarial content.
Use a dedicated virtual machine, container, or browser profile with minimal privileges. Mount only required files, limit network destinations, cap CPU and runtime, clear session data after the task, and separate test accounts from production accounts.
Do not expose a user's personal desktop, password manager, full email account, cloud console, or unrestricted filesystem to an experimental agent. Scope each session to the minimum data and applications needed for one workflow.
Record screenshots and actions only under an explicit privacy policy. Visual traces can capture personal data, access tokens, customer records, and confidential documents, so storage, redaction, retention, and reviewer access require control.
Confirmation should occur after the agent has prepared the exact action but before it executes. Show the target, values, consequence, source evidence, and whether the action can be reversed. A vague approval at the start of a long session is not consent for every later click.
Require approval for purchases, messages, account changes, file deletion, public publication, accepting terms, sharing sensitive data, and any action whose target or amount changed after earlier approval.
Some actions should be blocked entirely in the computer-use layer and exposed only through a narrower authorized service. This is especially useful for money movement, role changes, production deployment, and bulk deletion.
A final success message can hide unsafe navigation, accidental disclosure, repeated clicks, or an action performed on the wrong account. Evaluation should inspect the full trajectory: observations, target selection, policy decisions, actions, recovery, and final verification.
Build tasks across screen sizes, slow networks, pop-ups, changed labels, expired sessions, inaccessible elements, duplicate buttons, prompt injection, and partial completion. Track task success, unsafe-action rate, confirmation precision, step count, recovery rate, latency, and cost.
Prefer an API or structured browser interface when it exposes the required operation; visual computer use is valuable for applications without a reliable integration surface. Run it in an isolated browser profile or desktop with no ambient credentials, restricted downloads, controlled clipboard, bounded filesystem access, and network policy appropriate to the task.
Every observe-decide-act cycle should record the screenshot or accessibility evidence used, proposed action, target coordinates or element, expected state change, and resulting observation. Re-observe after navigation or layout changes instead of reusing stale coordinates. Treat text displayed on a page as untrusted content that cannot override policy.
Pause before submitting forms, sending messages, purchasing, deleting, changing permissions, solving authentication challenges, or exposing private data. Show the user the exact target and values. Detect unexpected domains, modal overlays, download prompts, instruction injection, and loops, then stop rather than improvising around a safety boundary.
The model proposes an action; deterministic policy decides whether it may run, needs approval, or must be blocked.
from dataclasses import dataclass
@dataclass(frozen=True)
class Action:
kind: str
domain: str
consequence: str
ALLOWED_DOMAINS = {"portal.example", "docs.example"}
CONFIRM = {"send", "purchase", "delete", "accept_terms", "share_data"}
def policy(action: Action) -> str:
if action.domain not in ALLOWED_DOMAINS:
return "BLOCK_DOMAIN"
if action.kind in CONFIRM or action.consequence == "irreversible":
return "REQUIRE_EXACT_CONFIRMATION"
return "ALLOW_ONE_ACTION"
actions = [
Action("read", "docs.example", "none"),
Action("send", "portal.example", "external_message"),
Action("type_password", "unknown.example", "credential_exposure"),
]
for action in actions:
print(policy(action))ALLOW_ONE_ACTION
REQUIRE_EXACT_CONFIRMATION
BLOCK_DOMAINThe policy operates on trusted runtime facts, not on claims found in the screenshot.
A target assertion prevents a stale observation from authorizing a click after the page has changed.
def can_click(observation: dict, expected: dict) -> bool:
return all(observation.get(key) == value for key, value in expected.items())
expected = {
"domain": "portal.example",
"page": "Draft invoice INV-8",
"button": "Submit for review",
"account": "Training tenant",
}
fresh_observation = dict(expected)
stale_observation = {**expected, "account": "Production tenant"}
print("Fresh:", can_click(fresh_observation, expected))
print("Stale:", can_click(stale_observation, expected))Fresh: True
Stale: FalseProduction verification should combine semantic UI state with backend checks for consequential actions.
0 of 2 checked
Try this next
0 of 3 completed
Use the most reliable signals available. Semantic DOM or accessibility information improves targeting, while screenshots reveal visual state. Consequential actions should verify both the intended UI state and the business outcome when possible.
Low-risk, reversible work may be pre-authorized inside strict bounds. Financial, legal, privacy, account, publication, and irreversible actions should require exact confirmation or remain behind narrower services.
Explore 500+ free tutorials across 20+ languages and frameworks.