GatewayRedbirds.com

ESPN lets you set custom scoring, but you have to create the league and then have enough people to invite.

Do any of the sites have these settings in the public leagues, or let you join a custom league without being invited?

i believe you can create a public league in yahoo that allows you to have custom scoring options. It's worth at least checking out.