Screencast.com isn't in the list from your screenshot where it says "you may embed from the following sites:", so it's not supported by default.
You can go here to see which ones are available for your instance of XenForo:
admin.php?bb-code-media-sites/
And you can use this example to see how it's done for say MetaCafe;
admin.php?bb-code-media-sites/metacafe/edit
And it seems you can add your own ones here:
admin.php?bb-code-media-sites/add
I haven't seen anybody on XenForo share theirs yet for screencasts.com.