Skip to content

Conversation

@StanChan03
Copy link
Collaborator

@StanChan03 StanChan03 commented Aug 21, 2025

Purpose

Refactors Reasoning Strategies to be more intuitive. Fixes #198 . Additionally, it fixes some of the inconsistencies with sem_extract particularly its output parsing

Test Plan

Created comprehensive tests to test Reasoning with different operators and a mix of reasoning strategies.

Test Results

Screenshot 2025-08-21 at 12 38 33 AM

(Optional) Documentation Update

TODO: update docs on reasoning strategy

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • [] New feature (non-breaking change which adds functionality)
  • [x ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
    --> change to reasoning api
  • Documentation update
  • Performance improvement
  • Refactoring (no functional changes)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, updating docstrings
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@StanChan03 StanChan03 marked this pull request as ready for review August 21, 2025 07:38
@StanChan03 StanChan03 requested a review from liana313 August 21, 2025 07:39
@StanChan03 StanChan03 changed the title [WIP] initial refactor of Reasoning Strategies initial refactor of Reasoning Strategies Aug 25, 2025
Copy link
Collaborator

@harshitgupta412 harshitgupta412 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of checking whether prompt_strategy is None, it will be better to just define a DEFAULT_PROMPT_STRATEGY (there are already default args in PromptStrategy, so PromptStrategy() will be enough)
The default argument will then be PromptStrategy() instead of None

user_instruction: str,
prompt_strategy: PromptStrategy,
operation_type: str = "filter",
) -> tuple[list[dict[str, Any]], list[Any], list[str] | None]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, this is supposed to basically generate demonstrations using the teacher model for the given data. In that case, it will be better to use sem_ops directly with teacher model instead of bootstrap functions

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will make sure that things are reusable and prevent double work in case some parameters change. (like adding a new prompt strategy or new prompts or new sem op)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improving reasoning strategy support

3 participants