Juan Jose,
Pascal is correct that you need to setup a dissector table, but you also need to set and register a "decode as structure" (decode_as_t) using register_decode_as.
Your "RTPS payload" should fall into one of 2 categories:
1. You have a "unique identifier" that determines how to dissect the (next) payload. For example "Ethernet" has a type, IP has a protocol, TCP/UDP have a port. In this case you set your dissector table up with the "unique identifier type" (typically numeric, but there are string and GUID examples). Then you need to be able to save that "unique identifier" during dissection to provide to decode as functionality (as part of decode_as_t structure). Your plugins will also call dissector_add_<type>() to register their "unique identifier". dissector_add_<type> calls dissector_add_for_decode_as() internally, so there isn't a need for plugins to call it explicitly. packet-ip.c is probably a good example to follow.
2. There is no unique identifier and payload must be determined by Decode As. You still need a dissector table and plugins will still need to call dissector_add_for_decode_as(), but "setting up the dissector table for decode as" has been made simpler by register_decode_as_next_proto(). See
https://code.wireshark.org/review/22575 for example uses.
