I built a tool for mobile and computer use using local and remote LLMs

Hacker News

Created a tool that lets you use LLMs to automate task across mobile (android) and computer. Currently, this uses screenshots and LLMs support for extracting screen UI elements effectively. This is still a work in progress and attempting to make this work with local models via Ollama (the code is in place with some issues). As of now, Gemini and GPT 4o works the best for finding UI elements and planning the task. Some examples that work as of now: 1. Use gmail and ask <friend>@example.com for lunch next saturday 2. Start a 3+2 chess game on lichess Working demos: https://github.com/BandarLabs/clickclickclick This improves the cost of one automation task from approx. $0.6 via Claude to: $0.06 - OpenAI 4o mini as planner + free Gemini flash 1.5 (15 calls/min) The Llama vision models will eventually make it 0.

Visit product View on Hacker NewsDec 18, 2024

Share card

Actual performance

2points

3comments

Did not reach leaderboard

Launch Intel predictions

Analyze your own launch →

Product HuntOn track for Day 1 leaderboard · Strong signals: claude, model, computer · Missing: mac, agents, macos

93%93% predicted probability of success on Product Hunt, based on ML models trained on real launch data.

best fitHighest predicted score across all platforms for this description.

Indie HackersFits the IH revenue-focused audience · Strong signals: created, gemini · Missing: supports, reddit linkedin, podcasting

89%89% predicted probability of success on Indie Hackers, based on ML models trained on real launch data.

TrustMRRFits verified-revenue profile · Missing: mobile apps, ios, personal

59%59% predicted probability of success on TrustMRR, based on ML models trained on real launch data.

Hacker NewsStrong engagement from HN community · Strong signals: llama, io · Missing: https docs, excited, just released

55%55% predicted probability of success on Hacker News, based on ML models trained on real launch data.

nativeThis product was originally launched on this platform.

AppSumoStrong fit for a featured deal · Strong signals: calls · Missing: plus, platform, intuitive

54%54% predicted probability of success on AppSumo, based on ML models trained on real launch data.

Acquire.comPre-revenue stage for this audience · Missing: arr, mrr, revenue

13%13% predicted probability of success on Acquire.com, based on ML models trained on real launch data.

BetaListMay not resonate with beta-testers · Missing: web3, chat, crypto

0%0% predicted probability of success on BetaList, based on ML models trained on real launch data.

Incorrect prediction on native model

Similar products

I built "computer use" before Anthropic, while using their API54%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

I built "computer use" before Anthropic, while using their API

Hacker News2

Open Computer Use83%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Open Computer Use

Hacker News9

CUA-S1 – A System One Model for Computer Use42%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

CUA-S1 – A System One Model for Computer Use

Hacker News90

Claude Computer use86%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Computer use for automating operations

Product Hunt+641Artificial Intelligence

Agent – A Local Computer-Use Operator for macOS51%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Agent – A Local Computer-Use Operator for macOS

Hacker News6

"Computer use" mcp for webapps and Electron apps33%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

"Computer use" mcp for webapps and Electron apps

Hacker News2

Polaris – Toolbox for AI computer use50%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Polaris – Toolbox for AI computer use

Hacker News1

AutoBrowser – Automate your browser with Claude Computer Use56%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

AutoBrowser – Automate your browser with Claude Computer Use

Hacker News4

Clevrr Computer95%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Computer use but with OpenAI and Gemini models

Product Hunt+195Open Source

Cua88%Launch Intel prediction score: how likely this product is to succeed on its source platform, based on its name, tagline, and description.

Docker for computer-use agents

Product Hunt+389Open Source

Browse all products Analyze your launch