Displaying all NFT media types in Blinks

sRFC
1

Displaying all NFT media types in Blinks

Summary

Adding “animation_url” and “category” to the blinks response would allow the unfurler/extension to include more exciting media in Blinks. (“category” coming from the metaplex nft standard)

This would greatly benefit digital artists sharing their art more and more of which is not just images. Especially generative artists who currently have no way to share their code based art live on social media.

Vr/models might be more challenging to implement afaik (would require using the model-viewer library)
But video and html should be pretty straight forward. The “category” would determine how the media is rendered, if not present just fall back to icon/image.

very basic/crude example would be:

{category === "html" && (
    <iframe
        src={animation_url}
        sandbox="allow-scripts allow-same-origin"
        referrerPolicy="no-referrer"
        style="pointer-events: none; width: 100%; height: 100%"
    />
)}
{category === "video" &&  (
    <video
        src={animation_url}
        autoPlay
        muted
        loop
        controls="false"
        style="width: 100%; height: 100%"
    />
)}

Concerns to discuss

  • Security risks with iframe/html
  • Website performance

I’m not a security expert so please correct me if I’m wrong. But to my knowledge sandboxing the iframe should prevent most attack vectors. We also eliminate risks by preventing user interactions with “pointer-events: none” or an invisible overlay on top of the iframe.

For performance, I wonder how difficult it would be to add an intersection observer to each Blink, and only render the media when its in view (or even only render the highest Blink when multiple are in view)?

There would always be an icon/image to fall back on in the case of errors loading the media.

Curious to know everyones thoughts!

1 Like
2

@c4b4d4 and @nickfrosty would love to get your input here!

3

I think HTML has some implications that it wouldn’t be easy to decide if it’s good or not to include.

It is safe as in what is being hosted in the iframe cannot read/write its parent, but is not preventing people making HTMLs that look like Twitter asking for their password :face_with_head_bandage:

On the code you shared I wouldn’t use allow-same-origin, to make it secure, I think having it set makes it be treated as in the “same-origin” in certain validations

3D would be pretty cool, I think model-viewer could work.

Hiding the element when not visible does help on performance (also adds complexity). I think Twitter in a normal screen size has ~15 pre-loaded tweets on its HTML

For the videos, on Twitter when you have multiple of them, if you had one playing and want to play another one, the first one stops and then the second one starts playing. Would catch it to keep a good UX.

1 Like
4

Regarding phishing attacks, I think that could easily be prevented with either “pointer-events: none” or a invisible div overlay to prevent user interactions.

For “allow-same-origin” I’m open to removing that if it creates a security risk. Although it would greatly reduce what could be displayed (speaking from personal experience, many of my html nfts fetch other nfts/images or fonts)

3D would be rad, I guess I was thinking that it would be asking a bit much to add the dependency, but if thats not a concern, then yes absolutely we should add that to the spec as well

2 Likes
5

I personally think the idea of more rich experiences could be interesting, at least displaying more than a simple image (even though this could be an animated gif already)

Video could be possible but depending on the file size and the streaming medium, this could get hard to handle and cache. Especially when using proxy services to help protect user’s privacy. Proxy streaming a video can be expensive. And some services do not actually allow streaming (I think you cannot stream videos off of arweave for some reason)

The animation_url idea could be useful, but even still today not all popular Solana wallets support this. So I suspect they would be unlikely to support it this way. I suspect it is due to some of the proxy/streaming issues I mentioned above.

For html, while this could be really cool I personally think it is too much of a security risk for people and honestly not likely to be supported. Even if in an iframe. Too many attack vectors, even if you get everything else right.

I am not familiar with VR models at all, would action-aware clients (like wallets and other extension) have to enable this individually?

6

hmm, yah hadn’t thought about the proxy aspect for video, figured it would all be client side.

Can you be more specific on what the security risks for a sandboxed iframe would be?

If that is an impassable issue, then perhaps we could come up with a standard with tight guardrails for uploading p5 projects directly, so that there’s never any raw html

2 Likes
7

I am personally not sure of the security issues with iframes, I just know there can be lots. So it throws up read flags in my head

2 Likes