SSH to your server as a root user:-
Modify DirectAdmin configuration file
Kindly search the line with a port name
and replace the port as you wish to set it and save :
And then restart the DirectAdmin
Then make sure to open port 4141 in server firewall and try to access DirectAdmin in a browser.