feat: UQ as Intrinsic by ink-pad · Pull Request #551 · generative-computing/mellea

ink-pad · 2026-02-17T03:45:40Z

Component PR

Add check_certainty() convenience function for the uncertainty intrinsic from ibm-granite/granite-lib-core-r1.0. This is a wrapper function (like check_answerability() in rag.py), not a new Component. Also extracts the shared call_intrinsic() helper into _util.py so both rag.py and core.py can use it without duplication.

Description

Link to Issue:

Implementation Checklist

Protocol Compliance

parts() returns list of constituent parts (Components or CBlocks)
format_for_llm() returns TemplateRepresentation or string
_parse(computed: ModelOutputThunk) parses model output correctly into the specified Component return type

Content Blocks

CBlock used appropriately for text content
ImageBlock used for image content (if applicable)

Integration

Component exported in mellea/stdlib/components/__init__.py or, if you are adding a library of components, from your sub-module

Testing

Tests added to tests/components/
New code has 100% coverage
Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Wire up the uncertainty intrinsic from ibm-granite/granite-lib-core-r1.0 with a high-level check_certainty() API. The intrinsic evaluates model confidence in its response given a user question and assistant answer. - Add check_certainty(context, backend) in core.py - Extract shared call_intrinsic() helper into _util.py - Update catalog to point uncertainty at granite-lib-core-r1.0 - Add test, example, and README entry Co-Authored-By: ink-pad <inkit.padhi@gmail.com>

Co-Authored-By: ink-pad <inkit.padhi@gmail.com>

github-actions · 2026-02-17T03:45:52Z

The PR description has been updated. Please fill out the template for your PR to be reviewed.

mergify · 2026-02-17T03:46:15Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

jakelorocco

Thank you for the contribution @ink-pad. It would be helpful if you could add an explanation of the motivation for making this change along with filling out the PR template.

Additionally, I left some comments trying to clarify the decisions made below along with some of the longterm consequences / trajectories of the intrinsics code.

jakelorocco · 2026-02-17T14:07:51Z

mellea/stdlib/components/intrinsic/core.py

+
+from ....backends.adapters import AdapterMixin
+from ...context import ChatContext
+from ._util import call_intrinsic as _call_intrinsic


This and rag.py are the only places where we explicitly use _call_intrinsic. If we want to rename the function to call_intrinsic, we should just do it now and change the API.

I didn't want to make many changes to rag.py hence kept it this way. Let me know if the new changes look good to you.

jakelorocco · 2026-02-17T14:14:18Z

mellea/backends/adapters/catalog.py

 _RAG_REPO = "ibm-granite/granite-lib-rag-r1.0"
 _CORE_REPO = "ibm-granite/rag-intrinsics-lib"
+_CORE_R1_REPO = "ibm-granite/granite-lib-core-r1.0"


I don't understand the difference between these repos. It looks like most of them contain almost the same adapters. Can you please add comments here explaining the difference between each catalog?

For instance, why is only the uncertainty intrinsic being updated here? It looks like that repo contains newer versions of the other intrinsics as well.

Additionally, should we not be using the older repos? The _CORE_REPO even seems to have a different folder structure with different models. It's counter-intuitive that an adapter may exist for a model / intrinsic but we wouldn't be able to look it up.

@frreiss: Could you please chime in regarding this?

A decision was made to change the layout of the repositories. The new required layout requires multiple new repositories. The old repositories will be deleted once we are certain that there is not active code using the old repositories. The existing versions of the "core" models are in old repositories that should be deleted once the new repositories come online and have been linked into Mellea.

Can you please leave a comment in the code indicating something along these lines so that future changes can have that context?

Drop the underscore prefix and alias — use call_intrinsic consistently across _util.py, rag.py, core.py, and test_rag.py. Co-Authored-By: ink-pad <inkit.padhi@gmail.com>

jakelorocco

@ink-pad, not sure if you ran / tested the code in uncertainty.py, but I am hitting errors when doing so. I believe it is caused by the different structure of the core-r1 repo:

Details

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 402, in hf_raise_for_status
    response.raise_for_status()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models/ibm-granite/granite-lib-core-r1.0/revision/main

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jake/code/mellea/docs/examples/intrinsics/uncertainty.py", line 26, in <module>
    result = core.check_certainty(context, backend)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jake/code/mellea/mellea/stdlib/components/intrinsic/core.py", line 20, in check_certainty
    result_json = call_intrinsic("uncertainty", context, backend)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jake/code/mellea/mellea/stdlib/components/intrinsic/_util.py", line 29, in call_intrinsic
    adapter = GraniteCommonAdapter(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jake/code/mellea/mellea/backends/adapters/adapter.py", line 135, in __init__
    config_file = granite_common.intrinsics.obtain_io_yaml(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/granite_common/intrinsics/util.py", line 176, in obtain_io_yaml
    lora_dir = obtain_lora(
               ^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/granite_common/intrinsics/util.py", line 130, in obtain_lora
    local_root_path = huggingface_hub.snapshot_download(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/_snapshot_download.py", line 245, in snapshot_download
    raise api_call_error
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/_snapshot_download.py", line 165, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2867, in repo_info
    return method(
           ^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/hf_api.py", line 2661, in model_info
    hf_raise_for_status(r)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/mellea/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 452, in hf_raise_for_status
    raise _format(RepositoryNotFoundError, message, response) from e
huggingface_hub.errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6995c359-7aed45806689cfb5352cd20b;a9140376-a2f3-43a5-937d-191e342e783b)

Repository Not Found for url: https://huggingface.co/api/models/ibm-granite/granite-lib-core-r1.0/revision/main.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated. For more details, see https://huggingface.co/docs/huggingface_hub/authentication

The other intrinsic examples still run so I think this is an issue with the new repository structure. My guess is that it's either our code is not specifying some sort of repository type to tell granite_common how to correctly grab the io.yaml file given the new file format. Let me know what you find and please reach out if large code changes are necessary for implementation. Thank you!

jakelorocco · 2026-02-18T13:47:20Z

mellea/backends/adapters/catalog.py

 _RAG_REPO = "ibm-granite/granite-lib-rag-r1.0"
 _CORE_REPO = "ibm-granite/rag-intrinsics-lib"
+_CORE_R1_REPO = "ibm-granite/granite-lib-core-r1.0"


Can you please leave a comment in the code indicating something along these lines so that future changes can have that context?

ink-pad · 2026-02-18T15:29:51Z

@ink-pad, not sure if you ran / tested the code in uncertainty.py, but I am hitting errors when doing so. I believe it is caused by the different structure of the core-r1 repo:

Details

The other intrinsic examples still run so I think this is an issue with the new repository structure. My guess is that it's either our code is not specifying some sort of repository type to tell granite_common how to correctly grab the io.yaml file given the new file format. Let me know what you find and please reach out if large code changes are necessary for implementation. Thank you!

@jakelorocco : I ran the tests locally before pushing, and everything looked good. It is probably failing here because the models have private visibility? https://huggingface.co/ibm-granite/granite-lib-core-r1.0

jakelorocco · 2026-02-18T15:33:43Z

@ink-pad, not sure if you ran / tested the code in uncertainty.py, but I am hitting errors when doing so. I believe it is caused by the different structure of the core-r1 repo:
Details
The other intrinsic examples still run so I think this is an issue with the new repository structure. My guess is that it's either our code is not specifying some sort of repository type to tell granite_common how to correctly grab the io.yaml file given the new file format. Let me know what you find and please reach out if large code changes are necessary for implementation. Thank you!

@jakelorocco : I ran the tests locally before pushing, and everything looked good. It is probably failing here because the models have private visibility? https://huggingface.co/ibm-granite/granite-lib-core-r1.0

Then we will have to wait for those models to be public to merge this.

inkpad and others added 2 commits February 17, 2026 03:18

Fix uncertainty description in README

610a836

Co-Authored-By: ink-pad <inkit.padhi@gmail.com>

ink-pad requested a review from a team as a code owner February 17, 2026 03:45

Merge branch 'main' into feat/uq

82e1e99

jakelorocco reviewed Feb 17, 2026

View reviewed changes

Rename _call_intrinsic to call_intrinsic

e2c438f

Drop the underscore prefix and alias — use call_intrinsic consistently across _util.py, rag.py, core.py, and test_rag.py. Co-Authored-By: ink-pad <inkit.padhi@gmail.com>

jakelorocco requested changes Feb 18, 2026

View reviewed changes

Conversation

ink-pad commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Component PR

Description

Implementation Checklist

Protocol Compliance

Content Blocks

Integration

Testing

Uh oh!

github-actions bot commented Feb 17, 2026

Uh oh!

mergify bot commented Feb 17, 2026

Merge Protections

🟢 Enforce conventional commit

Uh oh!

jakelorocco left a comment

Choose a reason for hiding this comment

Uh oh!

jakelorocco Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

ink-pad Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

jakelorocco Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

ink-pad Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

frreiss Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

jakelorocco Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

jakelorocco left a comment

Choose a reason for hiding this comment

Uh oh!

jakelorocco Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

ink-pad commented Feb 18, 2026

Uh oh!

jakelorocco commented Feb 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

ink-pad commented Feb 17, 2026 •

edited

Loading