New Symfony installer: the fastest way to start your Symfony project

Yesterday, the Symfony team introduced the new Symfony installer. Its main goal is to help developers to create Symfony projects faster.

Until now, installing Symfony to start a new project required a few steps:

  1. Download the code (zip file, git clone, etc.)
  2. Install vendors with “composer install”
  3. Tweak configuration settings

The installer tries to do this in one step. It downloads a compressed file with all the code, including the vendors directory, so you don’t need anything else to run Symfony for the first time.

Installing the installer

The Symfony installer is a PHAR file that runs on PHP 5.4+ (if you are still using PHP 5.3, upgrade!). To install it in your system:

Unix-based systems (Linux, Mac OS)

$ sudo curl -LsS http://symfony.com/installer -o /usr/local/bin/symfony
$ sudo chmod a+x /usr/local/bin/symfony

Windows

c:\> php -r "readfile('http://symfony.com/installer');" > symfony

Create a new project

To create a new project, just execute “symfony new [directory]“:

New project

This will create a new Symfony project using the latest stable version. To use a different version, the command accepts another parameter to use a different version. This parameter can be a branch (it will install the latest version for the branch), a specific version or even “lts” to use the most recent LTS (long term support) version.

Generating a demo project

The tool includes a command to generate a demo project following the official best practices, so you can learn from it.

To generate the project, just run “symfony demo”:

Generate demo project

Then, execute “php app/console server:run” to test the demo project using the PHP built-in webserver. The demo project should look like this:

Demo project

And this is the demo backend:

Demo backend

What about Composer?

Composer is not being replaced. At all. In fact, Composer is being used in the server to download all dependencies before generating the compressed file that you will use. Also, you will still need Composer to add new dependencies or upgrading Symfony.

Future work

They have been working hard during the last few months to create the installer. It works great but it needs some extra features that I am sure will be added in the future, as the first goal was to have a tool easy to use and working in multiple platforms.

  • HTTPS support: currently, the installer downloads compressed files using HTTP, instead of HTTPS. Using HTTPS is highly recommended to avoid man-in-the-middle attacks, as someone might be able to change the contents of the compressed file to inject malicious code.
  • Caching: if you install the same version of Symfony twice, it will be downloaded twice. Adding a cache layer would be interesting, as it would also allow to have some sort of “offline mode”. Have you tried to start a new Symfony project while you are in a train? :)
  • Reduce file size: depending on the developer machine, the installer chooses between two formats: zip and gzip (tar.gz). Much better compression ratios can be achieved with bzip2 and xz. It would make the installer even faster!

Under the hood

The installer uses the Symfony console component to create a command-line application. Then, to create a new project, it chooses the format of the compressed file that will be downloaded using Distill. As the GZIP file is smaller than the ZIP one, it checks if your machine can decompress GZIP files. Files are downloaded with Guzzle and decompressed using again Distill. Finally, it removes some files that are not useful for your project, like LICENSE, UPGRADE*.md and CHANGELOG*.md files, generates a proper random secret value using openssl_random_pseudo_bytes() if available and creates an appropiate .gitignore file.

Photo: “Speed…”, by Rami.

March 27 / 2015
Author Raul Fraile
Category PHP, Symfony, Tutorials
Comments 1 Comment

Upcoming Conferences

WeCamp

PHP New Zealand

PHP Summer Camp Croatia

PHPNE

MadisonPHP

brnoPHP

SymfonyLiveLondon

ZgPHP

PHPSouthAfrica

PHPNWUK

SymfonyLiveNYC

PHPForumParis

PHPARG

PHPWorld

TechMeetupUY

SymfonyConMadrid

Security tools for PHP projects

Security is getting more and more important, and the PHP community has been doing great improvements in this topic during the last few years. From better configuration settings to provide some level of security by default to frameworks providing functionality to avoid common attacks such as XSS, CSRF or SQL injection.

If you don't know that a bug exists yet, is it still a bug?

If you don’t know that a bug exists yet, is it still a bug?

There are dozens of measures we can take in order to prevent and minimize security issues, both in the server and in the project’s code:

  • Disable PHP modules that are not in use.
  • Restrict information leakage (don’t display PHP errors or extra information like PHP version)
  • Turn off remote execution (allow_url_fopen and allow_url_include should be Off)
  • Control the size of POST requests and uploaded files.
  • Control the maximum time and memory PHP can take to minimize DoS attacks.
  • Disable dangerous functions such as eval() or exec() if they are not going to be used.
  • Disable session IDs in URLs to prevent session fixation.
  • Include a token in your forms to avoid CSRF.
  • Properly escape the output to avoid XSS.

Recommended article: “Linux: 25 PHP Security Best Practices For Sys Admins”

So, our server and our code is ready. But, what about code that we don’t have full control of? Composer has dramatically changed the way we organize our projects, with many small packages and libraries, and this adds a new challenges for keeping our projects secure. The question is: is it safe to rely on code that others have written? are these package maintainers taking the same care as I do in my projects?

Well, any piece of software can have bugs, and obviously open source projects are not an exception. The good point is that security researchers, once they find a vulnerability, it is reported and added to a database of known vulnerabilities. We basically need to find a way to avoid using code with known vulnerabilities, and there are some interesting tools out there to help us.

Tools

The main goal of automatic tools is to let you know of security issues as soon as possible. Here, I present four tools to check PHP packages, configuration settings and PHP versions.

Security Advisories Checker

This service, created by SensioLabs (the company behind Symfony) checks your composer.lock file to find packages you depend on that have known security issues. These security issues are included in a database called Security Advisories Database, which is public and updated frequently.

There are three ways to use the service:

* Online checker: uploading the composer.lock file.
* CLI Checker: to check files from the command line.
* API: it can be integrated in your own projects.

For example, if we execute the CLI tool to check a project using Symfony 2.1 and an older version of Swiftmailer, we get a few vulnerabilities:

$ security-checker security:check composer.lock

Security Check Report
~~~~~~~~~~~~~~~~~~~~~

Checked file: /tmp/composer.lock

[CRITICAL]
2 packages have known vulnerabilities

swiftmailer/swiftmailer (v4.2.2)
--------------------------------

* Sendmail transport arbitrary shell execution

http://blog.swiftmailer.org/post/88660759928/security-fix-swiftmailer-5-2-1-released

symfony/symfony (2.1.x-dev)
---------------------------

* CVE-2014-4931: Code injection in the way Symfony implements translation caching in FrameworkBundle

http://symfony.com/blog/security-releases-cve-2014-4931-symfony-2-3-18-2-4-8-and-2-5-2-released

* CVE-2014-6072: CSRF vulnerability in the Web Profiler

http://symfony.com/blog/cve-2014-6072-csrf-vulnerability-in-the-web-profiler

* CVE-2014-6061: Security issue when parsing the Authorization header

http://symfony.com/blog/cve-2014-6061-security-issue-when-parsing-the-authorization-header

* CVE-2014-5244: Denial of service with a malicious HTTP Host header

http://symfony.com/blog/cve-2014-5244-denial-of-service-with-a-malicious-http-host-header

* CVE-2014-5245: Direct access of ESI URLs behind a trusted proxy

http://symfony.com/blog/cve-2014-5245-direct-access-of-esi-urls-behind-a-trusted-proxy

This checker can only detect vulnerabilities that are referenced
Disclaimer in the SensioLabs security advisories database. Execute this
command regularly to check the newly discovered vulnerabilities.
```

If you check your dependencies and get any vulnerability, please upgrade your packages as soon as possible.

Roave Security Advisories

The Roave team has created a Composer package called roave/security-advisories that prevents you from installing packages with known security issues by using an smart trick: it adds all packages with vulnerabilities to the `conflict` section, making them unavailable for your project.

    "conflict": {
        "contao/core": ">=2.11.0,<2.11.16|>=3.0.0,<3.1.0|>=3.1.0,<3.2.0|>=3.2.0,<3.2.7|>=2.11.0,<2.11.17|>=3.0.0,<3.1.0|>=3.1.0,<3.2.0|>=3.2.0,<3.2.9|>=2.11.0,<3.0.0|>=3.0.0,<3.1.0|>=3.1.0,<3.2.0|>=3.2.0,<3.2.19|>=3.3.0,<3.4.0|>=3.4.0,<3.4.4",
        "doctrine/dbal": ">=2.0.0,<2.0.8|>=2.1.0,<2.1.2",
        "doctrine/doctrine-module": "<=0.7.1|<=0.7.1",
        "doctrine/orm": ">=2.0.0,<2.0.3",
         ...
    }

So you only have to add this line to your `composer.json` file and say goodbye to those packages!

"require":{
    ...
    "roave/security-advisories": "dev-master"
}

Keep in mind that due to the nature of the problem, there are no releases or stable versions of the package, you need to rely on the dev-master branch, which is updated periodically. So the package is only suited for installation in the root of your deployable project.

To generate the list of packages, the same list as in the previous tool is used.

iniscan

iniscan is a tool created by Chris Cornutt that scans your php.ini file to check for possible security issues from your configuration settings. The list of tests that iniscan performs can be seen by executing “iniscan list-tests”. Currently, there are around 40 tests, some of them are listed below:

  • allow_url_fopen: (must be Off) It is recommended to not allow opening remote files.
  • allow_url_include: (must be Off) It is also highly recommended to not be able to include remote files.
  • display_errors: (must be Off) You should never display PHP errors in production.
  • expose_php: (must be Off) If is set to On, it adds a HTTP header with information about the PHP version. It is not recommended to expose additional information.
  • session.use_only_cookies: (must be On) It is not a good idea to allow passing session ids in URLs. If someone else intercepts the session id (it can also be predicted or guessed using brute-force) can use it to enter into the session.
  • session.cookie_httponly: (must be On) It makes cookies only accessible by the HTTP protocol, not by JavaScript. It helps to minimize the effects of a XSS attack.

After executing the command, you will get an output like this:

$ iniscan scan --fail-only
== Executing INI Scan [03.11.2015 08:11:40] ==

Status | Severity | PHP Version | Key | Description
----------------------------------------------------------------------
FAIL | WARNING | | session.cookie_domain | It is recommended that you set the default domain for cookies.
FAIL | ERROR | 5.2.0 | session.cookie_httponly | Setting session cookies to 'http only' makes them only readable by the browser
FAIL | WARNING | | session.hash_function | Weak hashing algorithms in use. Rather use one of these: sha224, sha256, sha384, sha512, ripemd128, ripemd160, ripemd256, ripemd320, whirlpool, tiger128,3, tiger160,3, tiger192,3, tiger128,4, tiger160,4, tiger192,4, snefru256, adler32, crc32, crc32b, fnv132, fnv164, joaat, haval128,3, haval160,3, haval192,3, haval224,3, haval256,3, haval128,4, haval160,4, haval192,4, haval224,4, haval256,4, haval128,5, haval160,5, haval192,5, haval224,5, haval256,5
FAIL | ERROR | 4.0.4 | session.cookie_secure | Cookie secure specifies whether cookies should only be sent over secure connections.
FAIL | WARNING | 5.5.2 | session.use_strict_mode | Strict mode prevents uninitialized session IDs in the built-in session handling.
FAIL | ERROR | 4.0.3 | allow_url_fopen | Do not allow the opening of remote file resources ('Off' recommended)
FAIL | WARNING | | display_errors | Don't show errors in production ('Off' recommended)
FAIL | WARNING | | expose_php | Showing the PHP signature exposes additional information
FAIL | WARNING | | post_max_size | Unless necessary, a maximum post size of 200M is too large
FAIL | WARNING | | display_startup_errors | Showing startup errors could provide extra information to potential attackers
FAIL | WARNING | | open_basedir | Restricting PHP's access to the file system to a certain directory prevents file-based attacks in unauthorized areas.
FAIL | WARNING | | upload_max_filesize | The max upload size should not be too high, to prevent server overload from large requests
FAIL | WARNING | | post_max_size | The max upload size should not be too high, to prevent server overload from large requests
FAIL | WARNING | | memory_limit | The standard memory limit should not be too high, if you need more memory for a single script you can adjust that during runtime using ini_set()
FAIL | WARNING | | xdebug.default_enable | Xdebug should be disabled in production
FAIL | WARNING | | disable_functions | Methods still enabled - exec, passthru, shell_exec, system, proc_open, popen, curl_exec, curl_multi_exec

23 passing
3 failure(s) and 13 warnings

pcc is a similar tool which also checks for PHP configuration settings.

versionscan

Unlike previous tools, versionscan does not check your dependencies or php.ini settings, it checks PHP itself to detect if known vulnerabilities are fixed or not. It can be really useful to execute in production environments. It was also created by Chris Cornutt.

It contains a local database of checks defined in a JSON file, so you will need to update the tool frequently to get updated lists of checks.

For example, if we execute the tool while using PHP 5.5.19 we get the following warnings:

$ versionscan scan
Executing against version: 5.5.19
+--------+---------------+------+------------------------------------------------------------------------------------------------------+
| Status | CVE ID | Risk | Summary |
+--------+---------------+------+------------------------------------------------------------------------------------------------------+
| FAIL | CVE-2015-0273 | 0 | Use after free vulnerability in unserialize() with DateTimeZone |
| FAIL | CVE-2014-9425 | 7.5 | Double free vulnerability in the zend_ts_hash_graceful_destroy function in zend_ts_hash.c in the ... |
| FAIL | CVE-2015-0232 | 6.8 | The exif_process_unicode function in ext/exif/exif.c in PHP before 5.4.37, 5.5.x before 5.5.21, a... |
| FAIL | CVE-2014-9427 | 6.4 | sapi/cgi/cgi_main.c in the CGI component in PHP through 5.4.36, 5.5.x through 5.5.20, and 5.6.x t... |
| FAIL | CVE-2014-9426 | 7.5 | The apprentice_load function in libmagic/apprentice.c in the Fileinfo component in PHP through 5.... |
| FAIL | CVE-2015-0231 | 7.5 | Use-after-free vulnerability in the process_nested_data function in ext/standard/var_unserializer... |
| FAIL | CVE-2014-8142 | 6.4 | Use-after-free vulnerability in the process_nested_data function in ext/standard/var_unserializer... |
+--------+---------------+------+------------------------------------------------------------------------------------------------------+

Scan complete
--------------------
Total checks: 315

Each line contains the CVE (Common Vulnerabilities and Exposures) ID associated to the vulnerability, the risk (a number between 0 and 10) and a quick summary of the issue. If you need more information about specific CVEs, you can look for them in the National Vulnerability Database.

By default, the tool checks vulnerabilities for the version contained in the `PHP_VERSION` constant, but it can also check other versions by using the “–php-version” argument:

$ versionscan scan --php-version=5.6.0
Executing against version: 5.6.0
+--------+---------------+------+------------------------------------------------------------------------------------------------------+
| Status | CVE ID | Risk | Summary |
+--------+---------------+------+------------------------------------------------------------------------------------------------------+
| FAIL | CVE-2014-9427 | 6.4 | sapi/cgi/cgi_main.c in the CGI component in PHP through 5.4.36, 5.5.x through 5.5.20, and 5.6.x t... |
| FAIL | CVE-2015-0232 | 6.8 | The exif_process_unicode function in ext/exif/exif.c in PHP before 5.4.37, 5.5.x before 5.5.21, a... |
| FAIL | CVE-2015-0273 | 0 | Use after free vulnerability in unserialize() with DateTimeZone |
| ...
+--------+---------------+------+------------------------------------------------------------------------------------------------------+

Scan complete
--------------------
Total checks: 315
Failures: 11

Conclusion

We have seen for different tools to help you to minimize security risks in your PHP projects. They are meant to detect known vulnerabilities as soon as possible, so you can take the appropriate measures.

Photo: “Just a boring traffic light”, by Tilemahos Efthimiadis

March 23 / 2015
Author Raul Fraile
Category PHP
Comments No Comments

Upcoming Conferences

WeCamp

PHP New Zealand

PHP Summer Camp Croatia

PHPNE

MadisonPHP

brnoPHP

SymfonyLiveLondon

ZgPHP

PHPSouthAfrica

PHPNWUK

SymfonyLiveNYC

PHPForumParis

PHPARG

PHPWorld

TechMeetupUY

SymfonyConMadrid

Symfony2 components overview: Stopwatch

It’s been a long wait, but we are back again with the Symfony2 components series. In the 12th post of the series, we cover the Stopwatch component. Even though is one of the smallest ones, that does not mean is not important, as plays a crucial role when we want to profile our code.

The Stopwatch component provides methods to measure execution time

The Stopwatch component provides methods to measure execution time

Installation

The recommended way of installing the component is through Composer:

{
    "require": {
        "symfony/stopwatch": "2.4.*"
    }
}

If you have never used Composer before check out our Composer 101 post.

The component

The Stopwatch component provides useful methods to measure the execution time of pieces of PHP code. In addition to parsing the output of the microtime() function for us, it allows tagging events, measure events in periods and get the max memory usage.

The component was added in Symfony 2.2, extracted from the HttpKernel component, so it was not written from scratch.

Simple example

Let’s see the component in action! In the following example, we try to measure the time spend by PHP to run the code sleep(1), which is obviously 1 second, but there is also some internal processing time such as looking for the function in the functions table:


use Symfony\Component\Stopwatch\Stopwatch;

$stopwatch = new Stopwatch();
$event = $stopwatch->start('test');
sleep(1);
$event->stop();
var_dump($event->getDuration()); // int(1030)
var_dump($event->getMemory()); // int(524288)

Here, we create a Stopwatch object, which returns a StopwatchEvent object when calling the start() method. Each of the events we want to measure have a unique name, in this example, ‘test’ is the event name. Then, after waiting a second, we stop the event and display both the duration in milliseconds and the maximum memory usage.

A much more useful example would be the following one. In this example, we create a function to calculate the nth number in the Fibonacci sequence. As you may know, the nth number in the Fibonacci sequence is the sum of the previous two numbers of the sequence, except the first two, which are 1. So, the complexity of calculating the nth number using a naive approach increases as the number gets bigger. If we want to measure the time spend to calculate the first 35 numbers of the series, would be as simple as this:

use Symfony\Component\Stopwatch\Stopwatch;

function fibonacci($n)
{
    if ($n <= 2) {
        return 1;
    } else {
        return fibonacci($n - 1) + fibonacci($n - 2);
    }
}

$stopwatch = new Stopwatch();
for ($i=1; $i<35;$i++) {
    $eventName = sprintf('Fibonacci %d', $i);
    $event = $stopwatch->start($eventName, 'fibonacci');
    fibonacci($i);
    $event->stop();

    printf("- %s: %dms\n", $eventName, $event->getDuration());
}

We get the expected results, being exponentially slower in every iteration.

- Fibonacci 1: 0ms
- Fibonacci 2: 0ms
...
- Fibonacci 15: 1ms
...
- Fibonacci 20: 16ms
...
- Fibonacci 25: 180ms
...
- Fibonacci 30: 2076ms
- Fibonacci 31: 3806ms
- Fibonacci 32: 5733ms
- Fibonacci 33: 8874ms
- Fibonacci 34: 15672ms
- Fibonacci 35: 24304ms

Periods

The lap() method allows us to measure the time for the same event in periods, which can be useful if what we want to measure is done is several steps. For example, we could measure the total time for reading, parsing and writing the results in another file, but divide it in different periods.

use Symfony\Component\Stopwatch\Stopwatch;
use Symfony\Component\Yaml\Yaml;

$stopwatch = new Stopwatch();

$event = $stopwatch->start('files');

// read file
$contents = file_get_contents('data.yml');
$event->lap();

// parse
$parsed = Yaml::parse($contents);
$event->lap();

// generate output
$output = '';
foreach ($parsed as $user) {
    $output .= $user['name'] . "\n";
}
$event->lap();

// write
file_put_contents('output.txt', $output);
$event->stop();

var_dump($event->getDuration());
foreach ($event->getPeriods() as $period) {
    var_dump($period->getDuration());
}

We get as an output:

int(5)
int(0)
int(4)
int(0)
int(1)

The first result is the total time, while the other four correspond to the four different periods. We can see that the most expensive ones are parsing the YAML file and writing the output to disk.

Sections

Sections are a nice way to group different events, even if they take place in separate parts of our code. For example, we may want to measure all the time spent in IO operations, so we can create a group named “io” and use openSection() and stopSection() to have a logical separation.

In the following example, we have two sections: “io” and “parsing”. Instead of using the actual functions, we call sleep() to make the example clearer.

use Symfony\Component\Stopwatch\Stopwatch;

$stopwatch = new Stopwatch();

// section io (1)
$stopwatch->openSection();
$stopwatch->start('read_file', 'read');
sleep(1);
$stopwatch->stop('read_file');
$stopwatch->stopSection('io');

// section parsing (1)
$stopwatch->openSection();
$stopwatch->start('parse_file', 'yaml');
sleep(1);
$stopwatch->stop('parse_file');
$stopwatch->stopSection('parsing');

// section io (2)
$stopwatch->openSection('io');
$stopwatch->start('write_file', 'write');
sleep(1);
$stopwatch->stop('write_file');
$stopwatch->stopSection('io');

echo "Category 'io'\n";
foreach ($stopwatch->getSectionEvents('io') as $event) {
    printf(" - %s: %d\n", $event->getCategory(), $event->getDuration());
}

echo "Category 'parsing'\n";
foreach ($stopwatch->getSectionEvents('parsing') as $event) {
    printf(" - %s: %d\n", $event->getCategory(), $event->getDuration());
}

Here, in the first two blocks, we open both sections, while in the third block, we open the previously used section “io”. Notice that we only provide the section name when closing it or if it already exists. Finally, we get each of the events for both sections and display their duration times:

Category 'io'
 - default: 2003
 - read: 1001
 - write: 1001
Category 'parsing'
 - default: 1000
 - yaml: 1000

As you see, there is an extra event, named “default”, which holds the sum of the times of all the events of the section.

Extending it

Unlike most Symfony2 components, the Stopwatch component it is really simple with only 3 classes (actually 4, but one of them is internal), and does not provide easy ways to extend it using interfaces and dependency injection, so the only way would be to use inheritance. Even with inheritance, the use of private properties makes it really hard and forces you to rewrite the component almost entirely.

It is nice that the component remains small and works most of the times, but having the option to extend it somehow would be cool for some use-cases, like having information of the number of opened files, cache status or database congestion. However, it is interesting to see how changing properties and methods to private in other components led to cleaner and better solutions.

Internals

Due to its simplicity, there is not much to say about how it works internally. To measure time uses the microtime() function, passing TRUE as an argument to get the result as a float representing the current time in seconds since the Unix epoch accurate to the nearest microsecond. To get the memory usage, the memory_get_usage() function is used, again passing TRUE as parameter to get the real size of memory allocated from system instead of just the one used by emalloc().

Who’s using it?

More info

Photo: Stopwatch, by Julian Lim

March 16 / 2015

Upcoming Conferences

WeCamp

PHP New Zealand

PHP Summer Camp Croatia

PHPNE

MadisonPHP

brnoPHP

SymfonyLiveLondon

ZgPHP

PHPSouthAfrica

PHPNWUK

SymfonyLiveNYC

PHPForumParis

PHPARG

PHPWorld

TechMeetupUY

SymfonyConMadrid

Website optimization tips

Websites are now richer than ever. They have high quality images, astonishing animations and responsive CSS, but this comes with a cost: your server will have to serve bigger images and text files, and your clients’ browsers will need to use more CPU cycles to process your site. In this post we will see how we can improve this without affecting the quality of the beautiful site you devised.

The Peregrine Falcon is considered the fastest member of the animal kingdom.

The Peregrine Falcon is considered the fastest member of the animal kingdom. A really “optimized” bird.

GZIP compression

Enabling GZIP compression is probably the easiest way to give a boost to our website, as we can have it up and running in a matter of minutes.

GZIP provides lossless compression, that is, we can recover the original data when decompressing it. It is highly recommended for text files such as HTML, CSS, JavaScript or XML, and as it is really fast, there is almost no overhead both in the server or the browser.

Check out our post on how GZIP works to see how to enable GZIP compression in Apache or nginx.

Image optimization

Optimizing images is crucial, as they represent around 60% in average of the total weight of modern websites, being JPEG the most used format:

formats

The first step to optimize images, even before using any tool, is to determine which is the most appropriate format for a specific image.

JPEG

JPEG images are usually suitable for photographs and performs worst results for small images or containing text or sharp lines.

There are a few tools to optimize JPEG images, but we will see only two: jpegoptim and jpegtran.

jpegoptim is a command-line tool available in most Linux repositories, as well as in Macports and Homebrew. Supports lossless and lossy compression.

$ jpegoptim aircraft.jpg
aircraft.jpg 467x312 24bit Exif ICC JFIF  [OK] 31623 --> 30810 bytes (2.57%), optimized.

If we ask jpegoptim to discard ICC profiles, IPTC markers, EXIF markers and comment markers, we can achieve greater compression:

$ jpegoptim --strip-all aircraft.jpg
aircraft.jpg 467x312 24bit JFIF  [OK] 31623 --> 19261 bytes (39.09%), optimized.

By default, the lossless optimization mode is enabled. So, previous transformations have not had cost in terms of image quality. We can enable the lossy mode and so have even higher compression ratios (reducing quality). In the following example, a quality of 80% is defined. Ratio is quite impressive but obviously the quality of the image is not the same:

$ jpegoptim --strip-all -m80 aircraft.jpg
aircraft.jpg 467x312 24bit JFIF  [OK] 31623 --> 14981 bytes (52.63%), optimized.

Determining the optimal value of the quality loss is a manual process, but we don’t have to be afraid of it. Usually, a quality loss of 5% – 10% is not perceived by the regular human eye. There are other situations where quality is not important, so we can have a bigger loss, like in small thumbnails.

The other tool, jpegtran, performs only lossless transformation of JPEG files. It also allows us to transform the image: rotate, scale, convert into grayscale, etc.

$ jpegtran aircraft.jpg &gt; aircraft_comp.jpg
$ ls -lh
-rw-r--r-- 1 raul  raul  31K Apr 22 10:56 aircraft.jpg
-rw-r--r-- 1 raul  raul  20K Apr 22 11:01 aircraft_comp.jpg

Many times, grayscale images used in websites are not really grayscale, but color images using only gray tones. Changing the image mode to grayscale we can achieve better results.

PNG

PNG is a raster graphics file format that supports lossless data compression. PNG was created as an improved, non-patented replacement for GIF. It is usually used for graphics, but is capable to handle an impressive number of image types with good results.

OptiPNG is a PNG optimizer that, without losing any information, reduces the size of PNG files. It works in two steps: first, preprocesses the pixels to make them more “compressible”, that are later compressed using LZ77. It provides quite a few options and optimization modes.

Executing it with default settings we get a 39.18% decrease in this image:

$ optipng image.png
** Processing: image.png
450x90 pixels, 4x8 bits/pixel, RGB+alpha
Reducing image to 8 bits/pixel, 37 colors (34 transparent) in palette
Input IDAT size = 7467 bytes
Input file size = 8375 bytes

Trying:
  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 4017
                               
Selecting parameters:
  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 4017

Output IDAT size = 4017 bytes (3450 bytes decrease)
Output file size = 5094 bytes (3281 bytes = 39.18% decrease)

As it provides different optimization levels (-o0 … -o7), we can try to get better ratios by using more time/memory.

GIF

GIF is a bitmap format usually used for simple animations, but its usage is decreasing in favor of CSS3 animations. It is still a widely used format to display funny cats though :)

If you like computer science history, the GIF part is quite interesting. From the “Unisys/CompuServe GIF Controversy” episode, that led to the “Burn all GIFs campaign”, to crazy stuff that has been done recently such as gifsockets, a library to provide real time communication using animated GIFs as a transport layer.

Going back to the purpose of the post, even though GIF images are usually really small, we can reduce them a bit. Gifsicle is a command-line tool for creating or editing GIF images, and since the animations consist of frames and some parts of the image don’t change from one frame to another, GIFsicle detects it and does not carry over the duplicate pixel information:

$ gifsicle -O2 image.gif > image_opt.gif
$ ls -lh
-rw-r--r-- 1 raul  raul  565K Apr 22 14:07 image.gif
-rw-r--r-- 1 raul  raul  548K Apr 22 14:07 image_opt.gif

WebP

WebP is an image format created by Google that provides lossless and lossy compression for images. The compression ratio is significantly better than PNG (26%) and JPEG (25-34%). Unfortunately is still not supported in all browsers. Firefox, Internet Explored and Safari still don’t support this format.

Favicon

A favicon is that little icon associated with a web page that is displayed next to the URL. This icon is usually placed in the server’s root directory with the name favicon.ico. Forgetting to include this icon affects the performance. As web browsers ask for it explicitly, if we don’t have a favicon our web server will have to look for it every time and return a 404 Not Found error. So, the recommendation is to have a small (less than 1K) and cacheable favicon.

ImageMagick can help us to create small favicons by reducing its size (16×16 should be enough) and number of colors. For example, to create a 16×16 favicon using only 4 colors from a PNG image:

$ convert -resize 16x16 -colors 4 favicon.png favicon.ico

We can increase the number of colors until our favicon looks ok. For example, I was able to create a ServerGrove favicon of 318 bytes (current one weights 678 bytes) using 16 colors and with similar quality.

Text files size reduction

As we discussed earlier, text files must be compressed with GZIP as we can achieve great compression ratios. Let’s see what we can do if we want to go even further:

JavaScript and CSS

There are two common techniques to try to reduce JavaScript and CSS files size: minification and combination.

By minifying a JavaScript/CSS file, we remove unnecessary bytes, such as extra spaces, line breaks and indentation.

YUI Compressor is a JavaScript and CSS compressor tool created by Yahoo!. It is a JAR file so we only have to download it and execute it with the Java runtime. For example, to minify the jQuery library:

$ java -jar yuicompressor.jar jquery.js > jquery.min.js
$ ls -lh
-rw-r--r-- 1 raul  raul  239K Apr 22 13:25 jquery.js
-rw-r--r-- 1 raul  raul  127K Apr 22 13:26 jquery.min.js

Another common technique is to combine different JavaScript/CSS files into 1 (1 of each type, of course). This has two benefits: less round-trips (requests) and possibly better GZIP compression ratios, as it works even better for larger files.

Finally, there are specific techniques to reduce the size of JavaScript or CSS files. For example, in CSS, we can convert “margin: 0px 0px 0px 0px” to “margin:0″, or “border-color: #ffeedd” to “border-color: #fed”. These optimizations are also done by YUI compressor. For JavaScript, variable names can be changed automatically for shorter ones.

HTML

HTML files can be minified, but is not an easy task at all, as there are tags such as <pre>, <textarea>, <script> and <style> that may be affected if we remove certain whitespaces. It is also important to note that can be a slow process, so doing it on the fly may not be the best idea, but for files minified once and served many times.

htmlcompressor is a Java-based command-line tool to minify HTML files. For example, the index page of the ServerGrove site can be reduced from 29K to 23K:

$ java -jar htmlcompressor.jar -o servergrove.min.html servergrove.html 
$ ls -lh
-rw-r--r-- 1 raul  raul  29K Apr 22 12:54 servergrove.html
-rw-r--r-- 1 raul  raul  23K Apr 22 12:55 servergrove.min.html

Remember to test the results as it is still an experimental technique.

Other files

Web fonts

Web fonts usage is growing steadily. According to HTTP Archive, 43% of websites are using them:

fonts

Web fonts size is determined by the number of glyphs, metadata and the used compression method. To make it more complicated, there are four different formats (woff, ttf, eot, svg), and none of them provides universal adoption. Services like Google Fonts can help us with this, and as we will see, it also provides ways to optimize the font size.

For example, the Open Sans webfont, as supports 20+ languages, weighs in at over 217K. If we are using only latin characters, we can ask only for that subset, reducing its size dramatically to 36K:

<link href="http://fonts.googleapis.com/css?family=Open+Sans&amp;subset=latin" rel="stylesheet" />

We can even download a web font containing only a few characters, something useful for fonts that are only used for a title for example:

<link href="http://fonts.googleapis.com/css?family=Open+Sans&amp;text=ServerGrove" rel="stylesheet" />

Flash

Unlike web fonts, the use of Flash in modern websites is going down, but still relevant, as 29% of websites still have at least 1 Flash file.

flash

Using vector graphics and avoiding embedded fonts can help to reduce the size of Flash files. Also, when using raster images, previous tips for compressing images are applicable too. Said that, my personal recommendation is to not use Flash unless is strictly necessary, something that is becoming less and less common with CSS3 and the new JavaScript APIs.

Caching

Caching is essential in today’s websites and applications. The basic idea behind caching is not to serve the same content to the same client twice. It is especially useful for resources that don’t change often, such as CSS/JavaScript files or images. With caching, we reduce the number of request our web server will have to handle, as well as the data transmitted. In HTTP, this is achieved by using HTTP headers.

HTTP/1.1 provides the following caching response headers:

  • Expires and Cache-Control: max-age: These headers specify how long the browser can consider the resource as fresh and use it instead of requesting it again. Once this time is expired, the browser will request the resource to the server.
  • Last-Modified and ETag: These headers follow a different approach, as are two ways to determine if the file we are requesting is the same as we have in the cache, and in case it differs, download it again.

For static assets, the recommended strategy is to set Expires to one year and the Last-Modified date to the last time the resource was changed.

Network optimizations

Reduce cookie size

Cookies must be as small as possible. The reason is that they are sent in the HTTP headers in every single request. No matter if we are requesting an HTML page, JavaScript file or even images, they are sent always. Using a different domain/subdomain for serving static assets can help as no cookies will be sent.

DNS lookups

Web browsers have a limit on the number of concurrent connections, so it is common to parallelize downloads across hostnames (domains or subdomains). While this can be good measure to load your site faster, if the number of hostnames is too long the time that the client spends to resolve the domain can affect negatively.

Avoid bad requests

404 Not found or 410 Gone HTTP responses come from unnecessary requests, which waste server time and resources. Looking for broken links and removing them is simple, so there is no excuse to leave them there.

Tools

It is important to be able to measure our page as soon as we are improving its performance. Automatic tools are of great help, as they report potential issues and give us indications in order to improve them. The most important ones are Google PageSpeed, YSlow and GTmetrix.

PageSpeed Module

mod_pagespeed and ngx_pagespeed are open-source modules created by Google for Apache and nginx. These modules speeds up websites by automatically applying some of the recommendations we have been talking about in this post, without having to modify our existing content or workflow.

Resources

Photo: Peregrine Stretching Wings, by Jerry Kirkhart

March 13 / 2015

Upcoming Conferences

WeCamp

PHP New Zealand

PHP Summer Camp Croatia

PHPNE

MadisonPHP

brnoPHP

SymfonyLiveLondon

ZgPHP

PHPSouthAfrica

PHPNWUK

SymfonyLiveNYC

PHPForumParis

PHPARG

PHPWorld

TechMeetupUY

SymfonyConMadrid

Sunshine PHP 2015

ServerGrove is a Silver sponsor of this year’s Sunshine PHP Developer Conference in Miami!

We’ll have a table setup and we’d love to meet you! Stop by, say “hi!” and let’s talk shop.

Oh, and we’ll be giving away 3 months of free VPS service to folks who visit us (VPS300).

The conference schedule looks great! Some of the talks that we’re looking forward to are:

  • Advanced Git Skills
  • Your Inner Sysadmin
  • Getting Started with Varnish
  • Unit Testing PHP Applications (you do have a test suite for your applications, right? right?)
  • Tuning Nginx And PHP-FPM The Right Way
  • REST API Best Practices
  • Dockerize All The Things!

We’re excited for the conference and we’ll see you all there!

February 04 / 2015
Author Avi
Comments 1 Comment

Upcoming Conferences

WeCamp

PHP New Zealand

PHP Summer Camp Croatia

PHPNE

MadisonPHP

brnoPHP

SymfonyLiveLondon

ZgPHP

PHPSouthAfrica

PHPNWUK

SymfonyLiveNYC

PHPForumParis

PHPARG

PHPWorld

TechMeetupUY

SymfonyConMadrid