store IDNs (internationalized domain names) in IDNA (ASCII) in our database, not in Unicode

I changed my mind. In 1bf8f199 I allowed Unicode domain names to go into the database. I thought that was nice because it's what the user *means*. But it's not how the web works. Web and DNS were working, but mail wasn't. Postfix (as shipped with Ubuntu 14.04 without support for SMTPUTF8) exists in an ASCII-only world. When it goes to the users/aliases table, it queries in ASCII (IDNA) only and had no hope of delivering mail if the domain was in full Unicode in the database. I was thinking ahead to SMTPUTF8, where we *could* put Unicode in the database (though that would prevent IDNA-encoded addressing from being deliverable) not realizing it isn't well supported yet anyway.

It's IDNA that goes on the wire in most places anyway (SMTP without SMTPUTF8 (and therefore how Postfix queries our users/aliases tables), DNS zone files, nginx config, CSR 'CN' field, X509 Common Name and Subject Alternative Names fields), so we should really be talking in terms of IDNA (i.e. ASCII).

This partially reverts commit 1bf8f199, where I added a lot of Unicode=>IDNA conversions when writing configuration files. Instead I'm doing Unicode=>IDNA before email addresses get into the users/aliases table. Now we assume the database uses IDNA-encoded ASCII domain names. When adding/removing aliases, addresses are converted to ASCII (w/ IDNA). User accounts must be ASCII-only anyway because of Dovecot's auth limitations, so we don't do any IDNA conversion (don't want to change the user's login info behind their back!). The aliases control panel page converts domains back to Unicode for display to be nice. The status checks converts the domains to Unicode just for the output headings.

A migration is added to convert existing aliases with Unicode domains into IDNA. Any custom DNS or web settings with Unicode may need to be changed.

Future support for SMTPUTF8 will probably need to add columns in the users/aliases table so that it lists both IDNA and Unicode forms.

CHANGELOG.md

@@ -8,6 +8,11 @@ Mail:
 
 * POP3S is now enabled (port 995).
 
+System:
+
+* Internationalized Domain Names (IDNs) should now work in email. If you had custom DNS or custom web settings for internationalized domains, check that they are still working.
+
+
 v0.08 (April 1, 2015)
 ---------------------
 

management/dns_update.py

@@ -397,26 +397,17 @@ $TTL 1800           ; default time to live
 """
 
 	# Replace replacement strings.
-	zone = zone.format(domain=domain.encode("idna").decode("ascii"), primary_domain=env["PRIMARY_HOSTNAME"].encode("idna").decode("ascii"))
+	zone = zone.format(domain=domain, primary_domain=env["PRIMARY_HOSTNAME"])
 
 	# Add records.
 	for subdomain, querytype, value, explanation in records:
 		if subdomain:
-			zone += subdomain.encode("idna").decode("ascii")
+			zone += subdomain
 		zone += "\tIN\t" + querytype + "\t"
 		if querytype == "TXT":
-			# Quote and escape.
 			value = value.replace('\\', '\\\\') # escape backslashes
 			value = value.replace('"', '\\"') # escape quotes
 			value = '"' + value + '"' # wrap in quotes
-		elif querytype in ("NS", "CNAME"):
-			# These records must be IDNA-encoded.
-			value = value.encode("idna").decode("ascii")
-		elif querytype == "MX":
-			# Also IDNA-encoded, but must parse first.
-			priority, host = value.split(" ", 1)
-			host = host.encode("idna").decode("ascii")
-			value = priority + " " + host
 		zone += value + "\n"
 
 	# DNSSEC requires re-signing a zone periodically. That requires
@@ -510,7 +501,7 @@ server:
 zone:
 	name: %s
 	zonefile: %s
-""" % (domain.encode("idna").decode("ascii"), zonefile)
+""" % (domain, zonefile)
 
 		# If a custom secondary nameserver has been set, allow zone transfers
 		# and notifies to that nameserver.
@@ -555,9 +546,6 @@ def sign_zone(domain, zonefile, env):
 	algo = dnssec_choose_algo(domain, env)
 	dnssec_keys = load_env_vars_from_file(os.path.join(env['STORAGE_ROOT'], 'dns/dnssec/%s.conf' % algo))
 
-	# From here, use the IDNA encoding of the domain name.
-	domain = domain.encode("idna").decode("ascii")
-
 	# In order to use the same keys for all domains, we have to generate
 	# a new .key file with a DNSSEC record for the specific domain. We
 	# can reuse the same key, but it won't validate without a DNSSEC

management/status_checks.py

@@ -246,7 +246,8 @@ def run_domain_checks(rounded_time, env, output, pool):
 def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains):
 	output = BufferedOutput()
 
-	output.add_heading(domain)
+	# The domain is IDNA-encoded, but for display use Unicode.
+	output.add_heading(domain.encode('ascii').decode('idna'))
 
 	if domain == env["PRIMARY_HOSTNAME"]:
 		check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles)
@@ -639,7 +640,6 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, rounded_time=Fal
 		if m:
 			cert_expiration_date = dateutil.parser.parse(m.group(1))
 
-	domain = domain.encode("idna").decode("ascii")
 	wildcard_domain = re.sub("^[^\.]+", "*", domain)
 	if domain is not None and domain not in certificate_names and wildcard_domain not in certificate_names:
 		return ("The certificate is for the wrong domain name. It is for %s."

management/templates/aliases.html

@@ -27,7 +27,7 @@
     <label for="addaliasEmail" class="col-sm-1 control-label">Alias</label>
     <div class="col-sm-10">
       <input type="email" class="form-control" id="addaliasEmail">
-      <div style="margin-top: 3px; padding-left: 3px; font-size: 90%" class="text-muted">You may use international (non-ASCII) characters, but this has not yet been well tested.</div>
+      <div style="margin-top: 3px; padding-left: 3px; font-size: 90%" class="text-muted">You may use international (non-ASCII) characters for the domain part of the email address only.</div>
     </div>
   </div>
   <div class="form-group">
@@ -98,8 +98,8 @@ function show_aliases() {
           n.attr('id', '');
 
           if (alias.required) n.addClass('alias-required');
-          n.attr('data-email', alias.source);
-          n.find('td.email').text(alias.source)
+          n.attr('data-email', alias.source_display); // this is decoded from IDNA, but will get re-coded to IDNA on the backend
+          n.find('td.email').text(alias.source_display)
           for (var j = 0; j < alias.destination.length; j++)
             n.find('td.target').append($("<div></div>").text(alias.destination[j]))
           $('#alias_table tbody').append(n);

management/web_update.py

@@ -89,7 +89,7 @@ def make_domain_config(domain, template, template_for_primaryhost, env):
 
 	# Replace substitution strings in the template & return.
 	nginx_conf = nginx_conf.replace("$STORAGE_ROOT", env['STORAGE_ROOT'])
-	nginx_conf = nginx_conf.replace("$HOSTNAME", domain.encode("idna").decode("ascii"))
+	nginx_conf = nginx_conf.replace("$HOSTNAME", domain)
 	nginx_conf = nginx_conf.replace("$ROOT", root)
 	nginx_conf = nginx_conf.replace("$SSL_KEY", ssl_key)
 	nginx_conf = nginx_conf.replace("$SSL_CERTIFICATE", ssl_certificate)
@@ -213,7 +213,7 @@ def create_csr(domain, ssl_key, env):
                 "-key", ssl_key,
                 "-out",  "/dev/stdout",
                 "-sha256",
-                "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain.encode("idna").decode("ascii"))])
+                "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain)])
 
 def install_cert(domain, ssl_cert, ssl_chain, env):
 	if domain not in get_web_domains(env):

setup/migrate.py

@@ -67,6 +67,35 @@ def migration_6(env):
 	basepath = os.path.join(env["STORAGE_ROOT"], 'dns/dnssec')
 	shutil.move(os.path.join(basepath, 'keys.conf'), os.path.join(basepath, 'RSASHA1-NSEC3-SHA1.conf'))
 
+def migration_7(env):
+	# I previously wanted domain names to be stored in Unicode in the database. Now I want them
+	# to be in IDNA. Affects aliases only.
+	import sqlite3
+	conn = sqlite3.connect(os.path.join(env["STORAGE_ROOT"], "mail/users.sqlite"))
+
+	# Get existing alias source addresses.
+	c = conn.cursor()
+	c.execute('SELECT source FROM aliases')
+	aliases = [ row[0] for row in c.fetchall() ]
+
+	# Update to IDNA-encoded domains.
+	for email in aliases:
+		try:
+			localpart, domainpart = email.split("@")
+			domainpart = domainpart.encode("idna").decode("ascii")
+			newemail = localpart + "@" + domainpart
+			if newemail != email:
+				c = conn.cursor()
+				c.execute("UPDATE aliases SET source=? WHERE source=?", (newemail, email))
+				if c.rowcount != 1: raise ValueError("Alias not found.")
+				print("Updated alias", email, "to", newemail)
+		except Exception as e:
+			print("Error updating IDNA alias", email, e)
+
+	# Save.
+	conn.commit()
+
+
 def get_current_migration():
 	ver = 0
 	while True: