public static function tagYourTag(array $tag, array $rendererStates, XenForo_BbCode_Formatter_Base $formatter)
// return some HTML (or a template object)
That's an incredibly simplistic example of what a BB Code callback should look like.
The file would be located in library/YourAddOn/Path.php
You can see how XenForo does its own BB Codes in XenForo/BbCode/Formatter/Base.php.