How to register a packet dissector for packets whose port are not pre-defined?

edit

You can use multiple ports for the dissector so that both ports (ABC and Y) would be registered.

taking from the example

local wtap_encap_table = DissectorTable.get("wtap_encap")
local udp_encap_table = DissectorTable.get("udp.port")

wtap_encap_table:add(wtap.USER15, p_multi)
wtap_encap_table:add(wtap.USER12, p_multi)
udp_encap_table:add(7555, p_multi)
udp_encap_table:add(7666, p_multi)
udp_encap_table:add(7777, p_multi)

thre .register_heuristic can do this job.

determine proto by custom logic, rather than port.

-- heuristic_checker: determine which dissector to use
local function heuristic_checker(buffer, pinfo, tree)
    -- check length
    length = buffer:len()
    if length < 4 then return false end

    -- check something..

    if true then
        -- use my dissector
        MyProto.dissector(buffer, pinfo, tree)
        return true
    else 
        return false
    end
end

-- registe to udp
MyProto:register_heuristic('udp', heuristic_checker)

ref:
https://mika-s.github.io/wireshark/lu...

This is not an uncommon scenario. Protocols like FTP and VoIP protocols have similar characteristics. For FTP its the opening of a data connection, after negotiation via the control channel on the well-known FTP server port. In VoIP there is the example of SIP, which uses SDP to negotiate the ports to which the audio data is to be sent with RTP.

The infrastructure in Wireshark to support this is the 'conversation'. It's defined by its endpoints (IP/proto/port tuple, with optional wildcards) and can dynamically associate a protocol dissector to such conversation, eg. based on what is negotiated in a control channel. The README.dissector file has more information on this and the FTP and SDP dissectors should be illustrative as well.