House of Leaves and Hopscotch, primarily, but I'm planning to read quite a few others to see how they're put together.
Seems like solid picks.
To me it's the kind of thing in which an attempt at a particular method it's what's inspiring the creator. Categorization as atypical is what comes later. So of course you would be unsure how to plan one out, because there is possibly no idea yet.
However, requiring additional participation from the reader beyond reading and turning pages linearly is at least a mission statement to begin with. According to Wikipedia, the author of
Hopscotch wanted the reader to feel like a "true co-conspirator."
Hopscotch (Cortázar novel) - Wikipedia
I could imagine a simple practical example: revision. That chapter where the POV spoke to Nancy? Nancy actually wasn't there that day. It could be a prompt for the reader to return with a more discerning eye, noticing that it was odd how different Nancy's manner of speaking had been in that chapter, try to compare it to other characters.
Attempting to determine who the POV characters are could be another example of basically demanding that the reader flip back and forth between chapters, possibly requiring notes in margins, that kind of thing. Likely with the objective of solving a mystery of some sort.
The possibilities are as endless as your imagination. Though I have one absolute answer for one of your questions:
...but I'm not so sure ... how to get started writing it.
Just start. This is a challenging undertaking for any artist. It demands particularly rough drafting. A lot is not going to work. Even write the story the normal way first if that helps, giving you something to craftily butcher.