Two Coins Bug

There’s an old riddle I heard while growing up that I used to hate.

You have two coins that add up to 30 cents, but one isn’t a nickel. What are the two coins?

The answer, of course, is a quarter and a nickel. Like many riddles, it comes down to an assumption we make when parsing the question. In this case, it’s absolutely true that one of the coins isn’t a nickel — but the other one is.

This riddle felt particularly salient to some work I was doing in WordPress this week, around the logic of MySQL queries and the WP_Meta_Query functionality.

As sometimes an example can explain a bit more than theoretical, consider the following:

Post ID: 50, has a title of "Fifty", and two separate rows in the postmeta table, both under the 'color' key -- 'red', and 'blue'.

Post ID: 51, has a title of "Fiftyone", and one row in the postmeta table, under the 'color' key -- 'red'

How would you query for all posts that have color of red, but do not have color of blue? My first attempt was something along the lines of this:

get_posts(
  array(
    'meta_query' => array(
      array(
        'key' => 'color',
        'compare' => '=',
        'value' => 'red',
      ),
      array(
        'key' => 'color',
        'compare' => '!=',
        'value' => 'blue',
      ),
    ),
  )
)

However, to my initial surprise, this returned both ID 50 and ID 51! So I pulled up the query to see why that could be, and it turned out the generated SQL looked something like:

SELECT *
FROM wp_posts
INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id )
INNER JOIN wp_postmeta AS mt1 ON ( wp_posts.ID = mt1.post_id )
WHERE ( wp_postmeta.meta_key = 'color' AND wp_postmeta.meta_value = 'red' )
AND ( mt1.meta_key = 'color' AND mt1.meta_value != 'blue' )
GROUP BY wp_posts.ID

(slightly simplified)

Well, you would ask, how does that return post ID 50? Well, because both of the joins to the meta table wind up finding the same meta! We say not to return a result with a meta key of blue, but SQL interprets that as find me any entry with a meta key that is not blue — and as post ID 50 also has a meta key that is red (which is not blue) we’re in business!

Anyway, finding the absence of something in the where clause is tricky — as the where clause specifies what you want to select. I’d looked and I’m pretty sure just using what’s already there in the WP_Meta_Query class it’s not possible to properly structure this, I’d done some work on it a decade ago, implementing the EXISTS and NOT EXISTS comparators.

So we could get the result we wanted, if we rewrote the above MySQL to:

SELECT *
FROM wp_posts
INNER JOIN wp_postmeta ON ( wp_posts.ID = wp_postmeta.post_id )
LEFT JOIN wp_postmeta AS mt1 ON ( wp_posts.ID = mt1.post_id AND mt1.meta_key = 'color' AND mt1.meta_value = 'blue' )
WHERE ( wp_postmeta.meta_key = 'color' AND wp_postmeta.meta_value = 'red' )
AND ( mt1.post_id IS NULL )
GROUP BY wp_posts.ID

The rough idea is that it will attempt to do a left join instead of an inner join, which could not turn up any results from the joined table — so we explicitly want to query for rows where a column that would have to be there — post_id — being NULL to ensure that there was nothing for it to join to. Thereby ensuring it NO HAZ any blue values.

So here’s what I wound up writing. It’s still rough, but functional thus far in my testing (so long as the NO HAZ isn’t the first parameter in the meta_query — as written the regex relies on the aliasing of the table name to parse it out). There’s probably some cleanup that it needs, however …

/**
 * An extra filter to let our custom `NO HAZ` meta_compare work!
 */
function no_haz_meta_compare( $sql, $queries, $type, $primary_table, $primary_id_column, $context ) {
	global $wpdb;

	foreach ( $queries as $key => $query ) {
		if ( ! empty( $query['compare'] ) && 'NO HAZ' === $query['compare'] ) {

			// If we've got a NO HAZ query, pull out the table alias it's using for the join.
			$matches = array();
			$search  = sprintf(
				'#\( (\w+)\.meta_key = \'%1$s\' AND (\w+)\.meta_value = \'%2$s\' \)#',
				preg_quote( $query['key'], '#' ),
				preg_quote( $query['value'], '#' )
			);
			if ( preg_match( $search, $sql['where'], $matches ) ) {
				$table_alias = esc_sql( $matches[1] );

				// And then rewrite the JOIN bit to do a LEFT join instead of inner, so it can check for null (absence)
				$sql['join'] = str_replace(
					array(
						sprintf(
							"INNER JOIN {$wpdb->postmeta} AS {$table_alias} ON ( {$wpdb->posts}.ID = {$table_alias}.post_id )",
							$matches[1]
						),
						sprintf(
							"LEFT JOIN {$wpdb->postmeta} AS {$table_alias} ON ( {$wpdb->posts}.ID = {$table_alias}.post_id )",
							$matches[1]
						),
					),
					$wpdb->prepare(
						"LEFT JOIN {$wpdb->postmeta} AS {$table_alias} ON ( {$wpdb->posts}.ID = {$table_alias}.post_id AND {$table_alias}.meta_key = %s AND {$table_alias}.meta_value = %s )",
						$query['key'],
						$query['value']
					),
					$sql['join']
				);

				// And now yeet the original WHERE bit and swap it to testing for `post_id` in the postmeta table being null -- or confirming that none were found.
				$sql['where'] = str_replace(
					$matches[0],
					"( {$table_alias}.post_id IS NULL )",
					$sql['where']
				);

			}
		}
	}

	return $sql;
}
add_filter( 'get_meta_sql', 'no_haz_meta_compare', 10, 6 );

which would permit the following to return only post id 51:

get_posts(
  array(
    'meta_query' => array(
      array(
        'key' => 'color',
        'compare' => '=',
        'value' => 'red',
      ),
      array(
        'key' => 'color',
        'compare' => 'NO HAZ',
        'value' => 'blue',
      ),
    ),
  )
)

The code would probably be a lot cleaner if it was in WP_Meta_Query proper, instead of regex parsing it all out — or if there was a filter in `WP_Meta_Query::get_sql_for_clause` to allow overriding — but hopefully this serves as a useful proof of concept for someone else hitting a similar issue down the road.

Jetpack Infinite Scroll for Single Posts!

Had a problem come up recently where folks wanted to keep engaging visitors on their website on a single post page — keep loading more posts afterwards when they kept scrolling.

I’ve heard of this before, and even seen some plugins accomplish it — for example, Infinite Post Transporter, by Tom Harriganwp.org / github — the codebase of which looks to be a modified version of Jetpack’s Infinite Scroll from about six years ago — (contemporary link).

So, I was curious to see how far I could go about getting close to what we needed just by playing with Jetpack’s own library, rather than duplicating a bunch of the code in a second plugin.

For anyone that wants to skip to the end and just get something to play with, here’s the gist that I’ve got the code shoved into for now.

First, a couple specifications we’re working with here:

  • I want to make this work on single post pages, specifically of the post post_type.
  • I don’t want to modify Jetpack files, or deregister / replace them with customized versions of the files. Filters / actions only.

So, skimming through the Jetpack Infinite Scroll codebase, there’s a couple conditionals we’re gonna need to short out to get things triggering on single post pages.

The bulk of the per-page code comes from The_Neverending_Home_Page::action_template_redirect() — this is the function that will register/enqueue Infinite Scroll’s scripts and styles, and set up footer actions to populate some javascript globals that specify state. However, for single posts, we’ll need to override two points.

		if ( ! current_theme_supports( 'infinite-scroll' ) || ! self::archive_supports_infinity() )
			return;

By default, single posts aren’t archives, and don’t work. So let’s hotwire it! Digging into The_Neverending_Home_Page::archive_supports_infinity() we can see that the whole response to that call is simply passed through a filter — `infinite_scroll_archive_supported`

		/**
		 * Allow plugins to filter what archives Infinite Scroll supports.
		 *
		 * @module infinite-scroll
		 *
		 * @since 2.0.0
		 *
		 * @param bool $supported Does the Archive page support Infinite Scroll.
		 * @param object self::get_settings() IS settings provided by theme.
		 */
		return (bool) apply_filters( 'infinite_scroll_archive_supported', $supported, self::get_settings() );

So to filter this, we’ll want to make sure we’re using the right conditionals and not filtering too many pages. In this case, is_singular( 'post' ) feels appropriate.

For brevity here, I’ll just be using anonymous functions, but in practice it’s likely far better to name your functions so other code can remove the filters if it becomes necessary.

add_filter( 'infinite_scroll_archive_supported', function ( $supported ) {
	if ( is_singular( 'post' ) ) {
		return true;
	}
	return $supported;
} );

Groovy, now we’ve got the function registering the scripts we care about. But there’s a second conditional later in the ::action_template_redirect() method that we also get snagged on — ::is_last_batch().

		// Make sure there are enough posts for IS
		if ( self::is_last_batch() ) {
			return;
		}

Fortunately, like our last example, The_Neverending_Home_Page::is_last_batch() is also filterable.

		/**
		 * Override whether or not this is the last batch for a request
		 *
		 * @module infinite-scroll
		 *
		 * @since 4.8.0
		 *
		 * @param bool|null null                 Bool if value should be overridden, null to determine from query
		 * @param object    self::wp_query()     WP_Query object for current request
		 * @param object    self::get_settings() Infinite Scroll settings
		 */
		$override = apply_filters( 'infinite_scroll_is_last_batch', null, self::wp_query(), self::get_settings() );
		if ( is_bool( $override ) ) {
			return $override;
		}

So again, we can just override it as we had before, only this case returning false:

add_filter( 'infinite_scroll_is_last_batch', function ( $is_last_batch ) {
	if ( is_singular( 'post' ) ) {
		return false; // Possibly retool later to confirm there are other posts.
	}
	return $is_last_batch;
} );

Great! So now we’ve got the scripts that do the work being output on our single posts page, but we’re not quite there yet! We need to change some of the variables being passed in to Jetpack’s Infinite Scroll. To modify those variables, we have — you guessed it — another filter.

The JS Settings are being output in The_Neverending_Home_Page::action_wp_footer_settings() (rather than being added via wp_localize_script), and here’s the filter we’ll be working off of:

		/**
		 * Filter the Infinite Scroll JS settings outputted in the head.
		 *
		 * @module infinite-scroll
		 *
		 * @since 2.0.0
		 *
		 * @param array $js_settings Infinite Scroll JS settings.
		 */
		$js_settings = apply_filters( 'infinite_scroll_js_settings', $js_settings );

so we’ll start with a generic action like this, and start customizing:

add_filter( 'infinite_scroll_js_settings', function ( $js_settings ) {
	if ( is_singular( 'post' ) ) {
		$js_settings['foo'] = 'bar'; // any we need to make!
	}
	return $js_settings;
} );

We can verify this change is in place by loading up a single post page, and searching for foo — confirming that it’s in the encoded string that’s being output to the page. So now we need to start looking at the javascript that runs and see what changes to inputs we may need to make.

First, we’ll need to make sure that the wrapper container we want to append our new posts to is the same as the ID that we use on archive pages. If it’s not, we just need to override that. In my case, I had to change that to main on the single post pages — which can be done like so:

$js_settings['id'] = 'main';

If your wrapper id is the same as archive pages, this can just be skipped. Here we can also override some other options — for example if we would rather change the verbiage on the load more posts button we could do

$js_settings['text'] = __( 'Read Next' );

or to switch from the button click, to autoloading on scroll we could do

$js_settings['type'] = 'scroll';

Now, if you tried what we have so far, you may notice that instead of getting new posts, you’ll likely see the same post loading over and over forever. That’s because of the parameters getting passed through as query_args — if you pop open your browser window to examine infiniteScroll.settings and look at the query_args param, you’ll likely notice that we’ve gotten the infiniteScroll.settings.query_args.name populated! This is getting passed directly to the ajax query, so when trying to pull up more posts, it’s restricting it to only the already queried post, as that’s from the query WordPress ran to generate the current url’s response.

So let’s just nuke it, and for good measure ensure we don’t inadvertently re-display the post in question.

$js_settings['query_args']['name'] = null;
$js_settings['query_args']['post__not_in'][] = get_the_ID();

Cool cool cool. So at this point, when you run the post, everything /should/ look about right, except for one small thing! You may see the url updating as you scroll to append /page/2/ to the path!

Unfortunately a lot of this functionality is hard-coded and I couldn’t find a good way to override it to update the url to — for example — the url of the post you’ve currently got in view, so I wound up doing the next best thing — nuking the functionality entirely.

	/**
	 * Update address bar to reflect archive page URL for a given page number.
	 * Checks if URL is different to prevent pollution of browser history.
	 */
	Scroller.prototype.updateURL = function ( page ) {
		// IE only supports pushState() in v10 and above, so don't bother if those conditions aren't met.
		if ( ! window.history.pushState ) {
			return;
		}
		var self = this,
			pageSlug = self.origURL;

		if ( -1 !== page ) {
			pageSlug =
				window.location.protocol +
				'//' +
				self.history.host +
				self.history.path.replace( /%d/, page ) +
				self.history.parameters;
		}

		if ( window.location.href != pageSlug ) {
			history.pushState( null, null, pageSlug );
		}
	};

As the js relies on rewriting the url via calling .replace() on a string, if we eat the token it searches for off the end, it’ll just wind up replacing the url with itself, and doesn’t look awkward. And so —

$js_settings['history']['path'] = str_replace( 'page/%d/', '', $js_settings['history']['path'] );

So, functionally this should get you most of the way there. The only other bit that I had added to my implementation was something that could trivially be done anywhere — Custom CSS in the Customizer, or theme files — but I wanted to hide Post Navigation links in my theme. So I just did a fun little trick of enqueueing my one line css tweak (with appropriate conditionals) like so:

add_action( 'template_redirect', function () {
	if ( ! class_exists( 'Jetpack' ) || ! Jetpack::is_module_active( 'infinite-scroll' ) ) {
		return;
	}

	if ( is_singular( 'post' ) ) {
		// jisfsp = Jetpack Infinite Scroll for Single Posts
		wp_register_style( 'jisfsp', null );
		wp_enqueue_style( 'jisfsp' );
		wp_add_inline_style( 'jisfsp', 'nav.post-navigation { display: none; }' );
	}
} );

This way, if someone disables either Jetpack or infinite scroll, the conditional trips and it stops hiding post navigation links.

I hope this has been somewhat useful to someone. It’s probably not at the point where I’d be comfortable packaging it up as a plugin for end-user installation, but if you’ve got a passing understanding of code and how WordPress works, it shouldn’t be difficult to re-implement for a client. If there’s any other tweaks on Infinite Scroll that you wind up finding and would like to suggest, please feel free to leave a comment below, and it’ll hopefully be useful to others. Cheers!

I’m Learning the Core Media Modal.

Disclaimer: This is basically me stream-of-thought’ing things as I’m learning the Core Media Modal’s codebase.  It’s my scratchpad, and I’m merely making it public in the hopes that it may be useful to someone else at some point in the future.  Some things are probably very wrong.  If I catch it, I’ll likely come back and edit it later to be less wrong.  If you see me doing or saying something stupid, please leave a comment, so I can be less stupid.  Thanks!

The media is written in Backbone, using the `wp.template` wrapper around Underscore templates for rendering.  If you want to really dive in depth, but don’t yet have a really solid understanding of Backbone, I’ve had several people recommend Addy Osmani’s book “Developing Backbone.js Applications” to me.  As luck would have it, it’s available for free online.

When exploring the code in WordPress, it looks like it’s best to do the investigating in the develop.svn.wordpress.org repository’s src directory (yes, develop.svn matches to core.trac — basically because legacy reasons and not wanting to change core.trac’s url when they changed core.svn over to be the Grunt’d version), before the build tools such as Grunt have a chance to run Browserify on it #.  If you try to read through the code on the GitHub mirror, you’re gonna have a bad time, as that doesn’t have the `wp-includes/js/media/` directory with the source files in it.

Browserify is a slick little tool in Node that bundles up a bunch of files, and puts them into a single file, so you can `require()` them in JS.  This makes them easier to work with in the source, and quicker to load in a browser.  WordPress has been using it to compile the Javascript for media since 4.2 (#28510), when the great splittening happened.  If this intrigues or confuses you, Scott Taylor has a great write-up on that ticket about the whys, hows, and whatnot.  It originally merged in at [31373] halfway through the 4.2 cycle.

Oh, and all the actual templates that are parsed and rendered by the views are in `wp-includes/media-template.php`

Okay, time to dig in.  (So that I’m not inadvertently writing a book, I’m going to split this into a series — but if you’d like to read them all, I’m dropping them in a tag.  You can find them all here.)

On Core REST API Authentication

Having an API is well and good, but if there are no ways for third-party apps to actually authenticate and use the API, it’s not very useful.

Background

While the framework for the REST API was merged into WordPress Core with the 4.4 release, the only means of using any endpoints that currently require authentication are what is known as ‘cookie authentication’ — that is, piggybacking off of the authentication cookies (plus a nonce) that WordPress Core sets in the browser when you log in traditionally to your WordPress site.  Unfortunately, that leaves the REST API as little more useful than the legacy `admin-ajax.php` file.

Fortunately, there are several authentication methods being worked on at the moment, in plugin form, for consideration of merging in to Core.

I’m heading up one of them, called Application Passwords.  In short, it lets a user generate as many single-application passwords as desired — one for each phone, desktop app, or other integration desired — and later revoke any application password whenever desired without affecting other applications or the user’s normal authentication.  The passwords are then passed with each request as Basic Authentication, encoded in the header of each request, as per RFC2617.

The other plugin is OAuth 1.0a authentication (spec).  Most OAuth usage across the internet is actually OAuth 2.0 — however, OAuth 2.0 requires HTTPS on the server side.  Ordinarily for most hosted services, this is not a problem.  However, for a distributed platform like WordPress, this is untenable due to the majority of sites not supporting HTTPS.  So an older, more complex specification — designed to not require HTTPS — had to be used.

For the record, I’m fully expecting to see an OAuth 2.0 plugin be built in the near future for use on sites that have a SSL certificate and support HTTPS.  However, that may not be very useful for app developers that want a ‘build once, run everywhere’ authentication method that will always be available.

Limiting Permissions

One of the discussions that came up with regard to Application Passwords is whether a REST API request that uses Application Password authentication should be able to modify Application Password endpoints.

Now, this is a very interesting question, and it can lead to many more questions — such as if an Application Password shouldn’t be usable to create or delete other Application Passwords, whether they should be allowed to do other user-administrative tasks (providing the relevant user has those permissions).  After all, if we’re preventing them from making a new Application Password, but they can just go and change the user’s actual password or email address, it’s a rather silly restriction.

So, there are several possiblilities.

First, you can just say “Any ways in to your account give full access to everything your account can do.  Be careful what applications and websites you give access to.” — the most basic, relatively easy to understand way.  Honestly, this is my preference.

Secondly, when creating a new Application Password or connecting a new client via oAuth, you could do something like … selecting what ‘role’ you’d like to give that connection.  For example, if your normal user account as an Administrator, but you’re connecting an app that’s intended just for writing blog posts, you may want to downscale your role for that authentication key to either an Author or perhaps an Editor.  An advantage to this is that it would be more cross-API — that is, it would work just as well with the legacy XML-RPC API, as with the new REST API.

This ‘role restriction’ method may be somewhat fragile, as it would need to only filter `current_user_can` and `user_can` — but only when checking the current user’s ID.  However, that may goof up some cron tasks that may run on the same request as the REST API request or other incendtal things.

Thirdly, we could do something REST API specific — either whitelist or blacklist REST API endpoints based on authentication key.  So, when either creating a new Application Password or authorizing a new OAuth connection, you would set rules as to what endpoints it can be used to hit.  Perhaps you’d want to allow all `wp/v2` namespaced endpoints, but no endpoints added by plugins to custom namespaces.  Or you want to only allow it to access core `posts` and `taxonomies` endpoints.  Or even something like allowing it to access anything but `plugins`, `themes`, `users`, or `options` endpoints.

The downside of this is that it won’t work with the legacy XML-RPC API and the user interface for it would likely be far more confusing for users.  It also could get problematic as permissions may vary for who can read `options` endpoints and who can write to them — or the like.  So then it may further complicate to allowing GET requests but not POST, PUT, DELETE requests to certain endpoints.

Your Thoughts?

In the end, I’m not sure what the best path forward is.  Maybe I’ve missed something.  But I am confident that we need to start paying more attention to authentication and permissions for the forthcoming REST API.  If you have any thoughts or remarks, please leave them in the comments.

 

 

Understanding Security Holes

Just finished my talk on Secure Code at WordCamp Philly 2015!  Thanks to everyone who came, here are my slides:

Please Don’t Hack Core

Especially to neuter Core’s ability to upgrade itself.

After I published this, I had someone from cPanel reach out to have a more in depth conversation than it’s really possible to manage in a medium that caps you at 140 characters.

https://twitter.com/cpanelcares/status/597886956945711107

In the interest of transparency and context — as well as showing cPanel’s efforts thus far in working to fix things, here’s the conversation that transpired on that ticket — #6489755 on their internal ticketing system.  Any modifications on my part are purely for formatting, as well as omitting names of customer support folks.


cPanel:

Are you using the cPAddons tool within the cPanel interface to install & manage WordPress? If so, then yes, we disable the auto-update functionality within the application so the updates can be managed from the cPanel interface itself. The way our cPAddons tool tracks software is not compatible with the way WordPress updates, hence why we disable the auto-updates so we can track it through cPAddons.

If you’re not using the cPAddons tool to install/manage WordPress and have concerns of us modifying the core of the application, please let me know.

Regards,


*********** ********
Technical Support Manager
cPanel Inc.

Me:

I’m a Core Contributor to WordPress, not a cPanel User. I was speaking up on Twitter because I learned through some Forum threads that y’all were doing some very problematic things — which I’m hoping to address here.

Just to make sure we’re talking about the same thing, the three changes that I’m aware of are specifically noted are:

get_core_updates()

Short-circuited with:

# cPanel override: Disable all core updates to prevent conflict with cPAddons.
return false;

core_upgrade_preamble()

Short-circuited with:

# cPanel override: Do not display the current or the latest version, because we've disabled updates.
return;

WP_Automatic_Updater::is_disabled()

Short-circuited with:

return true; // Force this functionality to disabled because it is incompatible with cPAddons.

(Please note that all my code references to WordPress core are aimed at the latest revision of the `master` branch on GitHub)

It looks like when you’re hacking core, you’re turning off not merely Automatic Updates (as you suggested prior), but all WordPress Updates as a whole. This is a Very Bad Thing. If you were merely disabling Automatic Updates, but still leaving the user with the ability to use WordPress’s very well established upgrade system, that would be something else entirely — and in fact is documented extensively here: https://make.wordpress.org/core/2013/10/25/the-definitive-guide-to-disabling-auto-updates-in-wordpress-3-7/
— and can be done by adding a single line to your generated wp-config.php file when installing WordPress:

define( 'WP_AUTO_UPDATE_CORE', false ); # Disables all automatic core updates:

Why do you feel the need to fully disable all updates from within WordPress and force users to use either cPanel or FTP exclusively to upgrade WordPress? Why can’t they work in conjunction with one another?

Clearly, users have been dismayed and shocked when their installs haven’t been notified of security point releases that are available as y’all have killed the `get_core_updates()` function. Many don’t even realize they may need to go into cPanel to upgrade their WordPress install, and so their installation is left at an outdated, insecure version that is incredibly vulnerable to exploit.

cPanel:

 Thanks for the followup. The WordPress management through cPAddons is quite old, and very well may have been in place prior to having define( 'WP_AUTO_UPDATE_CORE', false ); within the WordPress application. I’m uncertain of that as I do not know when WP introduced that function but from Googling it the oldest result I can find is from 2012.

That said, cPanel does in fact do a few things with cPAddons in regards to customers who have out dated versions:

  • Whenever WP releases a maintenance build that addresses security concerns, we react very quickly to get our software updated to be available to customers.
  • By default, we define that software managed/installed through cPAddons is automatically updated when a new update is available.
  • Based on the above information, if the server administrator leaves the defaults enabled, once WP introduced a maintenance releases that corrects security concerns and we’ve tested and updated our source for it, customers will receive the release automatically.
  • If the server administrator decides to disable automatic software updates, the end user and systems administrator will still receive notifications that their installation is out of date accompanied with steps on how to update their application.

With that, I can definitely appreciate the concern for making it as easy and automated as possible for users to get updates for their WordPress, there’s definitely more to the situation that solely disabling WordPress’ automated updates in the core.

I’ve submitted a case (#188545) with our developers to have the logic for disabling updates changed from it’s current behavior, to using define( 'WP_AUTO_UPDATE_CORE', false );.

If you have any other questions, feedback, or concerns, please don’t hesitate to let me know.

Me:

By default, we define that software managed/installed through cPAddons is automatically updated when a new update is available.

Clearly, that’s not the case in practice. As the user who discovered this remarked:

In my audit, it appears that — of 37 total WP-based websites on our server — we have 14 that have not updated to the latest version of WordPress. Of those 14, the oldest version is 3.9 (which 3 of 14 are running), and the newest is 4.1 (which 4 of the 14 are running).

Source: https://wordpress.org/support/topic/wp-updates-not-showing-latest-wp-version-available?replies=34#post-6612888

If cPanel wants to manually update sites to current releases, I’m fully, 100% in favor of that. It’s a solid step to a safer, more reliable web.

My issue is that y’all are preventing users from updating themselves via the existing WordPress infrastructure. There’s really no reason for the blocking of an existing stable upgrade system. If you just need some way for cPanel to be notified on core upgrades, it’s relatively trivial to set up a perhaps 20 line function that will notify cPanel when it happens — and that will even account for users manually updating their WordPress installation via FTP, which your current version seems as though it would break during.

Here’s a proof of concept:


<?php
/**
* Compares a stored version of the WordPress version to
* the current version.
*
* If the current version has changed, update the stored
* version and run custom code that notifies cPanel that
* an update has occurred.
*/
function cPanel_wp_version_check() {
$old_version = get_option( 'cPanel_wp_version' );
if ( $GLOBALS['wp_version'] !== $old_version ) {
$new_version = $GLOBALS['wp_version'];
update_option( 'cPanel_wp_version', $new_version );
// DO THINGS HERE TO NOTIFY CPANEL OF SOFTWARE UPGRADE.
}
}
add_action( 'init', 'cPanel_wp_version_check' );

Does that make sense?

Also — I’m not a lawyer, but distributing a forked version of WordPress and still calling it WordPress (rather than cPanelPress or something) may be a trademark violation. Forking is 100% fine under the GPL for copyright on the code, but may be problematic from a trademark perspective — as what you’re distributing isn’t actually WordPress, but rather a hacked up version. If that makes sense?

cPanel:

Thank you for contacting us regarding people’s experience using WordPress as distributed via our Site Software feature. This feature is a method of installing and managing various third party applications. Applications installed via Site Software are intended to be managed entirely within Site Software, thus in-application updaters are disabled. Allowing the in-application update to proceed will cause a conflict between the updated application and the Site Software, which can easily result in confusion. From this perspective what we are doing is no different from Debian and other Linux distros that distribute applications with in-application updaters.

We generally release the latest version of WordPress within 1 to 5 days of the latest WordPress update. At minimum server administrators are informed each night of all Site Software applications that need updated. It is up to user’s to configure their notifications within cPanel to receive such updates.

Within the Site Software user interface, users are able to upgrade all applications that are out of date. In the admin interface, a server admin can choose to upgrade all Site Software applications on the entire server.

Based upon what the Drumology2001 user reported on the forum it appears something is amiss on that server. We’d love to examine that server to determine why WordPress updates were not available to the user. Based upon the fuzzy dates used on the forum, and compared with our internal records, the 4.1.1 update was available to the Site Software system prior to the initial post. We’ll reach out to him to determine whether there is anything we can do there.

I’d also like to thank you for pointing us in the direction of the various ways to manage WordPress updates (https://make.wordpress.org/core/2013/10/25/the-definitive-guide-to-disabling-auto-updates-in-wordpress-3-7/). We are currently reviewing, and discussing, these to determine which fits the Site Software distribution method the best. Using a method within the application to disable updates is usually preferred.

One of our concerns is that in-application updaters are incompatible with the Site Software distribution method. There are various things that could happen due to updating a Site Software managed application outside of Site Software. At minimum it means that from that point onward the server admin, and potentially the user, will be informed of a software update that is no longer needed. At worst someone will force an update that results in corrupting the installed application.
Handling those in a way that reduces frustration for everyone, and keeps support costs down is important to us.

Based upon the experience of the three users that posted to that thread (Drumology2001, echelonwebdesign, and sg2048) it is apparent there is room for other improvements within the system, such as update notifications. We’re taking into consideration their experience to determine how we can making WordPress hosted on a cPanel & WHM server a better experience for all.

Me:

 To summarize, my argument is largely a pragmatic one.

You can’t prevent a user from updating the software outside of the cPanel Site Software Distribution Method (gosh, that’s a mouthful, I’ll call it the CPSSDM from here on out) — as they could always just use FTP to update the software.

From a software architecture perspective, this is problematic — and it would be far simpler to develop around it so that any software updates run — whether via FTP, a remote management tool (such as ManageWP, InfiniteWP, Jetpack Manage, iThemes Sync, etc), via the WordPress Dashboard, or an Automatic Update — successfully apprises the CPSSDM of the update.

Long story short, WordPress updating itself, and the CPSSDM managing updates shouldn’t be a conflict, they should behave in concert with one another, complementing each other in their behaviors, rather than stomping across the sandbox to kick over the other’s castle.

As mentioned above, I’d be delighted to volunteer my time and expertise to help the CPSSDM have an integration that doesn’t involve hacking core files and potentially leaving users running insecure software.

cPanel:

Thanks for the follow up and summary. We both agree there are improvements to be made, as with any software – it’s never ending. We will definitely reach out directly once your expertise is needed to make sure we’re providing the absolute best experience we can to both of our customers.

Thank you again, and keep that feedback coming!


So, in then end I don’t know where things are going from here.  I know that a lot of users find it super convenient to use one click installs for WordPress, and I really hope that users who take the short-cut of one-click installs don’t wind up dead ended on an old insecure release because of some sort of server misconfiguration and hacked core files.

I’m also optimistic, because cPanel seems willing to take suggestions and input from the community on best practices.  After all — let’s face it, when they’re providing one click installs for dozens of software projects, they’re not going to be able to work with each software project individually to make sure they’re doing it the best way possible.  A lot is dependent on the communities reaching out and offering to help them do it the right way.

I look forward to hearing back from cPanel and seeing their integration done in a way that works well and plays nice with everyone else in the sandbox too.

After all, I think we all want sustainable, stable integrations — not fragile bits of code that will break if a user upgrades the wrong way. 🙂

Why Jetpack isn’t a Collection of Plugins, Part the First

In keeping with a previous post I’d made a couple months ago explaining the oft-discussed rationale of why we do things the way we do with Jetpack, I’ll be doing it again today, on a different — but related — topic.

I may as well make a series of it.

This is the first of two posts (in theory, I’ll remember to write the second) explaining why Jetpack is a big plugin with many features, rather than many individual plugins.  This post will be looking at the primary technical reason.  The abundance of other reasons will be in the subsequent post.  (So please don’t read this post and think it’s the only reason — it’s not)

tl;dr: Dependency management sucks.

Jetpack, as you may be aware, is structured as a bunch of modules.  Many — but not all — require a connection to WordPress.com to function.  This isn’t for vanity purposes, it’s because they actually leverage the WordPress.com server infrastructure to do things harder, better, faster, stronger than a $5/month shared host is capable of.  To do that, they need to be able to communicate securely with WordPress.com, and WordPress.com must be able to communicate securely back to your site.

Some of the modules that require a connection are things such as Publicize (which uses the WordPress.com API keys to publicize to assorted third-party systems, rather than making users register various developer accounts and get their own API keys), Related Posts (which syncs some content up to the WordPress.com servers and indexes it on a large ElasticSearch index more efficiently and accurately than could be done in a MySQL database), Monitor (which pings your site every five minutes and emails you if it’s down), Comments (which passes data back and forth behind the scenes to enable secure third-party comment authentication) — you get the idea.

We could bundle the connection library with each individual plugin.  However, we’d need to make sure it was namespaced correctly so each different plugin can use its own correctly versioned instance of the connection classes.  Which would then mean a user could have well over a dozen copies and different versions of the same connection class active at a given time.  Which will make things more difficult with respect to developing the plugins, as you can’t assume methods in one are necessarily in another.  And when you make a change in the master class, you need to scan each repository to make sure you’re not breaking anything there, and keep changes synced to well over a dozen repositories.  But I digress.

To avoid duplicate code, the modules that depend on talking back and forth with WordPress.com all use a common library that handles signing and verifying requests, API calls, and the like.

Because it’s all packaged in a single plugin, we can be sure that it’s all running the required version.  If Publicize needs a change in the core connection library, we can be sure that the version of the connection library in Jetpack has those changes.  If the core connection library needs to change structure, we can make sure that any modules that used the old methods are updated to run the new ones instead.  Everything is maintained so that it’s running smoothly and works properly with each other.

Now, if Likes, Single Sign On, After the Deadline, Post by Email and others were their own plugins, and connected to a separate Jetpack Core plugin, versioning gets tricky.  It could work, in theory, if every plugin is kept up to date, always and forever.  But the instant that the user is using, say, an outdated version of Subscriptions with an outdated Jetpack Core (which work perfectly together), and then installs the up-to-date WP.me Shortlinks plugin, things could break because WP.me Shortlinks expects a more up-to-date Jetpack Core.  So you go ahead and update Jetpack Core to current, but now Subscriptions — which used to work perfectly — now breaks because there was a method change in Jetpack Core, that is fixed in the up-to-date version of Subscriptions, but the user isn’t running the up-to-date version.  Horrible UX.

Plus, if the user doesn’t have any Jetpack stuff, the installation flow for their first Jetpack Plugin that needs the core would be something like this:

  1. Install Stats.
  2. Activate Stats.
  3. Get error saying you need Jetpack Core for Stats to function.
  4. WTF is Jetpack Core? I just want Stats!
  5. Okay, install Jetpack Core.
  6. Activate Jetpack Core.
  7. Wait, what was I doing?
  8. Stats!  Okay, right.
  9. Connect the Jetpack Core to WordPress.com.
  10. Drink Scotch in celebration.

Compare this to the status quo of:

  1. Install Jetpack.
  2. Activate Jetpack.
  3. Connect Jetpack to WordPress.com.
  4. Stats is already active (unless you’re Mark Jaquith, in which case you activate it in this step)
  5. Drink Scotch in celebration.

As I said, dependency management is hard, and there’s not really a good way to manage it in WordPress.  There have been some very worthwhile attempts made, but none that can have a sufficiently solid user experience for an average user to compare with our current system and flow.

Any questions or suggestions about dependency management and Jetpack? Ask away!

Better handling of RTL stylesheets in plugins and themes

Or, How I Learned To Stop Worrying and Let Core Do The Work For Me.


If you’ve ever done something like this, we should have a little chat:

if ( is_rtl() ) {
	wp_register_style( 'example', plugins_url( "css/example-rtl.css", __FILE__ ) );
} else {
	wp_register_style( 'example', plugins_url( "css/example.css", __FILE__ ) );
}
wp_enqueue_style( 'example' );

Now, don’t worry. I’ve done it too! It’s not that big of a deal. But there is a better, tidier way. Just compare the above code blurb to this:

wp_register_style( 'example', plugins_url( "css/example.css", __FILE__ ), array(), '1.0' );
wp_style_add_data( 'example', 'rtl', 'replace' );
wp_enqueue_style( 'example' );

which will output something like this in rtl locales:

<link rel='stylesheet' id='example-rtl-css' href='http://domain.com/path/to/css/example-rtl.css' type='text/css' media='all' />

Simpler, right? It reads more easily, and as an added bonus, if something is to toggle RTL after you’ve registered the path to the asset, it handles it gracefully! As it doesn’t determine which asset path to serve up until it’s actually outputting the tag.

Now, this is assuming that your rtl stylesheet is just a replacement for your normal stylesheet. Which most are — it could be automatically generated with some tool like CSSJanus or CSS-Flip. But if you’ve got an add-on css file, that you want to load in addition that just contains overrides for RTL languages, you can handle that just as easily!

wp_register_style( 'example', plugins_url( "css/example.css", __FILE__ ), array(), '1.0' );
wp_style_add_data( 'example', 'rtl', 'addon' );
wp_enqueue_style( 'example' );

which will output something like this in rtl locales:

<link rel='stylesheet' id='example-css' href='http://domain.com/path/to/css/example.css' type='text/css' media='all' />
<link rel='stylesheet' id='example-rtl-css' href='http://domain.com/path/to/css/example-rtl.css' type='text/css' media='all' />

For the curious as to how Core actually does it, read here:

https://github.com/WordPress/WordPress/blob/809baf442b/wp-includes/class.wp-styles.php#L88-L104

Detailed explanation (with bonus examples for handling minified versions of both regular and rtl css as well):

/**
 * If you're supplying a pre-minified version of the stylesheet, you'll
 * need this, and to add the `suffix` data, so that core knows to
 * replace `example-min.css` with `example-rtl-min.css` -- handling
 * the suffix properly.
 */
$min = ( defined( 'SCRIPT_DEBUG' ) && SCRIPT_DEBUG ) ? '' : '.min';

/**
 * The normal registration. You're familiar with this already.
 */
wp_register_style( 'example', plugins_url( "css/example{$min}.css", __FILE__ ), array(), '1.0' );

/**
 * I set the value to 'replace', so it will replace the normal css file if rtl,
 * but it could also be 'addon' for a css file that just gets enqueued as
 * well, rather than replacing the normal one.
 */
wp_style_add_data( 'example', 'rtl', 'replace' );

/**
 * Finally, if we are replacing the existing file, and there's some sort of
 * suffix like `-min` as mentioned earlier, we need to let core know about
 * it, so that it can keep that suffix after the added `-rtl` that it's adding to
 * the path.
 */
wp_style_add_data( 'example', 'suffix', $min );

/**
 * Then we just enqueue it as we would normally!  If it's going to always
 * be enqueued regardless, we could just call `wp_enqueue_style()` rather
 * than `wp_register_style()` above.
 */
wp_enqueue_style( 'example' );

On Jetpack and Auto-Activating Modules

Hopefully, this is the last time that I’ll have to answer this question.

Frankly, it’s been answered dozens of times before. Now, I’m hoping to use this as a canonical ‘Answer Link’ that I can refer people to.  I’ll keep up with comments, so if anyone would like to ask

So, why does Jetpack auto-activate features?

Well, to start off, I should probably clarify what we currently do on this. We don’t auto-activate every new module that comes in.

We never auto-activate features that affect the display or front-end of your site — or at least not unless a site administrator explicitly configures them to.

So, for example, something like Photon, which would swap all your content images to CDN-hosted versions, doesn’t auto-activate. Our comments system doesn’t auto-activate either, as that would swap out your native comment form. Our sharing buttons do, but they don’t display unless you take the time to drag down some sharing buttons to the output box under Settings > Sharing.

However, modules like Publicize, Widget Visibility, and the like — they just give you new tools that you can use, with no risk to affecting your everyday visitors. When users upgrade, we give them a notification of what just happened, and point out some new features we’ve built in that they may want to activate themselves.

One thing we’ve recently expanded on, perhaps six months ago, is a ‘plugin duplication list’, for lack of a better phrase. These aren’t plugins that have an actual code-based conflict with a module, they’re ones that may be … duplicating effort. Previously, we were just scanning for plugins that would output OG Meta Tags, and short-circuit our own provider. However, since Jetpack 2.6, which shipped in November 2013, we’re actually doing it via a filter for all modules. For example, if you’ve got Gravity Forms or Contact Form 7 installed and active, our internal Jetpack Contact Form won’t auto-activate. If you’ve got AddThis or ShareThis active, our sharing buttons module won’t even kick in.

Now, obviously, we can’t catch every single plugin that may be similar enough to one of our modules to give cause to negate auto-activation. So there’s a filter, `jetpack_get_default_modules`, that can be used in any plugin to cancel auto-activation on any module.


add_filter( 'jetpack_get_default_modules', 'my_jetpack_get_default_modules' );
function my_jetpack_get_default_modules( $modules ) {
    return array_diff( $modules, array( 'module-slug' ) );
}

But I don’t like auto-activation of new features!

Okay.

You’re totally allowed not to.

We’re going to continue using our discretion to auto-activate select modules by default, but if you’d like to turn it off permanently for yours or a client’s site, we’ve made it ridiculously easy to do.


add_filter( 'jetpack_get_default_modules', '__return_empty_array' );

That’s it.

We believe that judiciously enabling new features is a win for users, especially considering 1) how low-impact most features are when ‘active’ but not actually implemented by a site owner, 2) how awkward it is for a site owner to have to enable something twice — for example, enabling the Custom Post Formats bit, and then having to visit Settings > Writing in order to actually enable the Portfolio custom post type.

We’ve spoken to many, many users who find a default feature set convenient, and resent having to make a bunch of ‘decision points’ if they had to manually activate each and every module. Good software should run well out of the box. So we’ve set up the defaults as we have. Yes, some people disagree and are vocal about not wanting anything to auto-activate. That’s okay. We try to design for the majority, with the best user experience we can provide.

If you have clients, that you’d like to be active in the relationship with, and customize the Jetpack experience for — that’s terrific. You’re the type of people that we add bunches of filters for. We’re all about empowering you to override our decisions, we just prefer to keep the default user interface free of a thousand toggles.

Decisions, not options — no?

Twitter oEmbed is Leaking Data

oEmbeds are fun.  They make it easy to embed third-party content on your site, like tweets, status updates, videos, images, all sorts of stuff.

Unfortunately, to do this, third-party code gets injected into your page.  Don’t worry, this is by design, but it does mean that you should only oEmbed from reputable sites.  WordPress Core is very picky as to the providers that it chooses to accept as oEmbed sources.

Twitter is one of these oEmbed providers.  Here’s an example of an embedded tweet:

Neat, isn’t it?

Now, hover over my name.

See that little url that shows on the bottom left corner of your browser (probably)?  It probably looks just like http://twitter.com/daljo628!

Now, click it.  Don’t worry, I’ll wait.

Did the page you landed on have a bunch of extra cruft appended to the end of it?

Maybe it looked something like https://twitter.com/daljo628?original_referer=http%3A%2F%2Fstephanis.info%2F2014%2F05%2F22%2Ftwitter-oembed-is-leaking-data%2F&tw_i=469505591946129408&tw_p=tweetembed?

If you right-click and inspect the element, the URL is just what you expected! If you right-click and open in a new tab — same thing! But if you click normally and let it trigger a Javascript event, it modifies the link before your browser actually processes it.

After you’ve clicked on it normally once, you can come back and re-inspect it, to see that the URL on the link has now changed to the one with the referer data on it — they’re rewriting it inline and intentionally delaying it so when you first click, you wouldn’t realize that the data was being appended.

This can be a problem because some sites employ concealers for the referer http header (No, I didn’t misspell referrer) like href.li for example. By embedding this in a get parameter forcibly, it’s leaking data in a way very difficult to block, by taking advantage of the trust offered via accepting Twitter as an oEmbed provider.