[ad_1]
Running LLaMA and Gemma LLMs with C++ and Python
Nowadays, “mobile AI” is a fast-growing trend. Smartphones become more powerful, and large models become more efficient. Some customers may want to wait until new features are added by phone manufacturers, but can we use the latest AI models on our own? Indeed, we can, and the results are fun. In this article, I will show how to run LLaMA and Gemma large language models on an Android phone, and we will see how it works. As usual in all my tests, all models will run locally, and no cloud APIs or payments are needed.
Let’s get into it!
Termux
The first component of our test is Termux, a full-fledged Linux terminal made as an Android application. It is free, and it does not require root access; all Linux components are running exclusively in a Termux folder. Termux can be downloaded from Google Play, but at the time of writing this text, that version was pretty old, and the “pkg update” command in Termux did not work anymore. A newer version is available as an APK on the F-Droid website; it works well, and I had no problems with it.
When Termux is installed on the phone, we can run it and see a standard Linux command-line interface:
In theory, we can enter all commands directly on the phone, but typing on the tiny keyboard is inconvenient. A much better way is to install SSH; this can be done by using “pkg install”:
pkg update
pkg upgrade
pkg install openssh
After that, we can start the SSH daemon in Termux by running the sshd command. We also need to get the user name and set the SSH password:
sshd
whoami
#> u0_a461
passwd
#> Enter new password
...
Now, we can connect to a phone with any SSH client:
ssh -p 8022 u0_a461@192.168.100.101
Here, 8022 is a default Termux SSH port, “u0_a461” is a user name that we get from a “whoami” command, and “192.168.100.101” is the IP…
Source link
