As an alternative to running LLMs on your desktop you can signup to Groq AI for free calls.
If you don’t have a beefy PC + GPU combo, Groq AI is what you what to run llama3 70B.
Advantages
- Right now it is free for everyone, no billing details even required.
- Groq AI inference times are even quicker than GPT4
- Access to llama3 70B (the largest model)
- Recent info, knowledge cutoff date is Dec 2022 for 70B and March 2023 for 8B vs SCM AI Jan 2022
Disadvantage
- Low rate limits. You wont be able to run more than 1 AI thread
- Rate limit wait times can be long. SCM is able to wait between calls to maximize throughput though
Get Groq
Create a free account
Get your API key
Click the create api key button
Paste key into SCM
API logins
Select default ai = groq ai
Paste in API key
Test
