Your stream, at the very least, is only as good as what you’re sending to Owncast. So make sure you’re sending from your broadcasting software at a quality that you’re happy with. Read through our Broadcasting Software Suggestions where some specifics are listed for your software.
You may also want to increase the amount of CPU being used to process the video on your Owncast server. As you go lower you will visibly see the quality degrade. However, make sure your server can handle more CPU being used or you’ll experience different problems.