Claude/configure llama local ai 011 cv5d q jn6 v yp dc cabn ex xf

[37-AN]

Summary by cubic

Adds local Llama support via Ollama, including a new provider, Docker profile, and setup docs. Airweave now prefers local models with automatic cloud fallback for generation, embeddings, and reranking.

• New Features

  • Added OllamaProvider with text generation, structured JSON, embeddings, and reranking.
  • docker-compose: new ollama service under the local-llm profile with healthcheck and persistent volume.
  • Config/env: OLLAMA_BASE_URL in Settings and .env; factory initializes Ollama when base URL is set.
  • defaults.yml: added Ollama models and set provider order to prioritize Ollama across operations.
  • Documentation: QUICK_START_OLLAMA.md and LOCAL_LLAMA_SETUP.md.

• Migration

  • Start services: docker-compose --profile local-llm up -d.
  • Pull models: llama3.3:70b (or llama3.2:3b) and nomic-embed-text.
  • Set OLLAMA_BASE_URL in .env (http://localhost:11434) or use the compose default.
  • Optional: remove cloud API keys to force local-only; enable GPU by uncommenting the compose GPU section.

^{Written for commit 7ee0e79. Summary will update automatically on new commits.}

[cubic-dev-ai]

1 issue found across 8 files

Prompt for AI agents (all 1 issues)

Understand the root cause of the following 1 issues and fix them.


<file name="backend/airweave/search/factory.py">

<violation number="1" location="backend/airweave/search/factory.py:687">
Rule violated: **Check for Cursor Rules Drift**

This change introduces a new Ollama provider, but `.cursor/rules/search-module.mdc` still documents only Cerebras, OpenAI, Groq, and Cohere. Please update the Cursor rule to include Ollama’s capabilities, configuration keys (e.g., `OLLAMA_BASE_URL`), and operation preferences so Cursor users get accurate guidance.

(Based on your team&#39;s feedback about verifying Cursor rules per provider.) [FEEDBACK_USED]</violation>
</file>

---

Since this is your first cubic review, here's how it works:

• cubic automatically reviews your code and comments on bugs and improvements
• Teach cubic by replying to its comments. cubic learns from your replies and gets better over time
• Ask questions if you need clarification on any suggestion

_{React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

[cubic-dev-ai]

Rule violated: **Check for Cursor Rules Drift** This change introduces a new Ollama provider, but `.cursor/rules/search-module.mdc` still documents only Cerebras, OpenAI, Groq, and Cohere. Please update the Cursor rule to include Ollama’s capabilities, configuration keys (e.g., `OLLAMA_BASE_URL`), and operation preferences so Cursor users get accurate guidance. (Based on your team's feedback about verifying Cursor rules per provider.)

View Feedback

Prompt for AI agents

```text Address the following comment on backend/airweave/search/factory.py at line 687: This change introduces a new Ollama provider, but `.cursor/rules/search-module.mdc` still documents only Cerebras, OpenAI, Groq, and Cohere. Please update the Cursor rule to include Ollama’s capabilities, configuration keys (e.g., `OLLAMA_BASE_URL`), and operation preferences so Cursor users get accurate guidance. (Based on your team's feedback about verifying Cursor rules per provider.) @@ -678,6 +680,11 @@ def _init_all_providers_for_operation( + ctx.logger.debug( + f"[Factory] Attempting to initialize OllamaProvider for {operation_name}" + ) + provider = OllamaProvider(base_url=api_key, model_spec=model_spec, ctx=ctx) if provider: ```

[cubic-dev-ai]

No issues found across 1 file

[omgupta-iitk]

Hi @37-AN ,

Are you currently working on this PR? Integrating Ollama into Airweave is a great idea, but after testing the changes, it seems the implementation is still incomplete. I’m running into several issues, and it would have been better to open this as a draft PR if it’s still a work in progress. Also, as a general practice, it’s important to test changes locally before submitting a PR. It looks like some of the code may have been pushed directly after being generated by an AI model, which can lead to incomplete or incorrect, especially in a public repository.

I also checked your other PR about fixing the 384-dimension issue in the dense embedding class. That could be a valuable improvement as well, since currently only the OpenAI embedder is fully supported. However, for the Ollama integration (or any non-OpenAI embedding setup) to work end-to-end without external APIs, the same embedding model should be used for generating vectors for both queries and entity data. This isn’t implemented yet in either PR. The errors you’re seeing are consistent with that mismatch.

Additionally, there seems to be an issue with the Ollama search service implementation. The vector size mismatch suggests that the dimension is defaulting to 384 somewhere. You’ll likely need to track down where that default is coming from and set the embedding size for nomic-embed-text correctly so that the search pipeline works as expected.

Thanks

[37-AN]

Thanks for the detailed feedback and for taking the time to test the changes thoroughly. You’re absolutely right—I should have opened this as a draft PR since the implementation isn’t complete yet. I apologize for not testing more comprehensively locally before pushing.

Regarding your points:

1. Draft PR Status: I’ll convert this to a draft immediately to reflect that it’s still WIP.
2. Embedding Model Consistency: You’ve identified the core issue perfectly. The mismatch between query embeddings and entity embeddings is causing the problems. I need to ensure that when Ollama is used, the same model (nomic-embed-text) generates vectors for both the query pipeline and the entity data ingestion.
3. Vector Dimension Issue: The 384-dimension default is likely hardcoded somewhere in the pipeline. I’ll trace through the embedding configuration to find where nomic-embed-text’s actual dimension (768 for nomic-embed-text) needs to be explicitly set, rather than falling back to the OpenAI default.
4. Testing Before Submission: Lesson learned—I should have run the full end-to-end flow locally with actual Ollama models before opening the PR. I’ll make sure to do thorough integration testing going forward.

I’m actively working on addressing these issues. Would it be helpful if I outlined my plan for completing the implementation in the PR description? I want to make sure this adds real value to Airweave once it’s properly finished.

Thanks again for the constructive feedback—it’s much appreciated.