Automate: Generating webshops in 7 seconds

After announcing my plans to start working for myself to my employer a couple months ago, I went from full-time to part-time employment in mid-October. Since then, I’ve gotten a lot more productive and I’ve gained a lot of time to work on projects and ideas I’ve had for months or even years. In the past weeks I’ve soft launched Webtrimmer, a Software as a Service web-application which improves and compresses the technical aspect of websites and images with one click of a button and I launched a new webshop which I hadn’t touched since April. Two weeks ago I made a small list with projects to work on each week, last week I worked on a way to easily create affiliate sites.

About a year ago I bought four different websites, because I wanted to learn the aspects of different online businesses. I bought a webshop for smartphone cases with it’s own stock built with PrestaShop, an informational website about dog tax with AdSense ads built with WordPress, a New Age-related webshop with dropshipping built with a Webshop as a Service-platform and a watches-webshop with an affiliate-feed built with WordPress. The affiliate site was a big disappointment because I didn’t know that the product feeds were specific to each affiliate user, the WordPress-site was slow even without any content and I needed a plugin for the insane amount of €299 without knowing I would ever earn that back with this site. Even though I bought 4 domains with good SEO-names to set up more affiliate sites I never even signed up to any Affiliate Network.

Over the next couple of months I had to focus on my full-time employment, also because I did my graduation internship there and I wanted to get my degree this year. After successfully graduating I felt very relaxed and free, but it took me only a couple of days to realize that I wanted to work for myself more than anything else. This has been in the back of my mind for the last 10 to 15 years but having no business experience, over-thinking every single little thing and afraid to make any mistakes always led to paralyzing fear to even start the smallest form of a business, even though I learned a lot in theory. Having saved quite a bit of money allowed me to risk losing my job without having to worry about income for some months, so I talked to my employer about what I wanted to do. My employer was very open to the idea of me becoming a part-time employee which gave me a lot more time to start setting up my own business and generating an income from that. For the past two years I talked about travelling to Asia as soon as I got my degree, but now I’m using my savings for an entirely different adventure. I plan to do everything online so as soon as this adventure proves to be sustainable, I can just pack my laptop and work from anywhere in the world!

Back to the topic of this article. After buying multiple domains for affiliate sites and a very slow WordPress site without any content the idea to make a Site Generator started to grow inside my head. The Affiliate Network has a generic product-feed for every single publisher, so I figured that generating multiple sites based on unique feeds wouldn’t be hard to do. I decided to create an application that generates HTML-files, because the content is static for every single visitor and I don’t have to deal with stock-changes. Combining this with some AJAX and JSON allows for really fast loading times, with fully-loaded pages of 40+ products and images within 150ms on my own machine. I chose not to use any sort of framework, so a non-minified, non-gzipped page without images is on average no more than 10 kilobytes.

Design

The design was something I wanted to keep very minimalistic, mainly because that’s my preferred style but also because it would be a lot easier to only worry about one accent color. For the navigation bar I decided to take inspiration from my own webshop, because I wanted a design that could handle 4 navigation-items, or 40.

This is what it looks like on my webshop:

And this is what it looks like for the affiliate site, dark red is the accent color:

For the description pages I took inspiration from a bunch of different ecommerce websites, eBay, Amazon, AliExpress, Coolblue.nl and Bol.com, which ended up looking like this:

Development

On Saturday the 12th of November I started with a simple piece of code to retrieve the product-feed so I could test the different variables that were available:

After that I started creating a template for the different variables and to test the look and feel of the website without having to re-generate the entire site for every single change. I set up the template in such a way that I could use the exact same files to generate all the static HTML-files and to make visual changes in the layout with default variables. I decided to split the product-feed on brands because I figured that a brand was something every single product in every single feed would have. Every brand would become a navigation-item and every one of those pages would get it’s own JSON-feed, to prevent loading the entire product-feed for every single category and to easily show all products on those pages without having to filter the data while a visitor is waiting for the products to show up. The more I can calculate and generate beforehand, the faster the final website will be. After creating all the CSS and a config-file I triggered the generator and this was the result after some testing:

Awesome! It only took about 7 seconds to generate all the files for a website with over 800 different products and pages. Now I wanted to generate more of these sites and the Generator worked flawlessly for the other feeds, most of my time was spent picking an accent color and the right product-feed. Which resulted in this:

All of this was done in a single day, I had done so much and it was only Saturday. This took me about 7 hours, starting with nothing more than a vague idea and ending with multiple fully-generated, decently-designed, user-friendly, super-fast websites.

Sunday

The next day was a lot less productive, the fun part was over. I was mainly fixing all kinds of inconsistencies with the product-feeds, like having brands in all-caps or hyphenated, some product names having special characters, or some products having no brand at all. After fixing all those things I decided to be more productive the next day.

Monday

Monday I started with the sitemap.xml, it generates an entry for every single page including the homepage and every detail-page. I also added a “focus”-subcategory, which allows me to use an extensive feed but only use the products from the “smartwatch” category for example. This gave me the idea to have the Generator create pages and categories based on those categories instead of just the brands. That way I could have a “smartwatch”, “smartphone” and “laptop” category that all have the brand “Apple” instead of one “Apple”-category with a random mix of products in it. This would also allow me to more easily combine product-feeds from different publishers, because I can then use a product-feed for smartphones from Site A and a feed for headphones from Site B. I wrote these ideas down and moved on, because I don’t think I have enough time left this week to include those now.

Wednesday

Tuesday I worked on several of my other projects, adding a payment provider to my webshop and creating some ads and posts on social media. Wednesday I decided that I didn’t want to hotlink to the external images from the feeds, mainly because they were oversized, uncompressed and loading those images will always be slower than from a local server. I created a component that checks if the image-file already exists on the local server and if not will download, resize and compress the image. Resizing and compressing the images saved on average about 80% of the total size and it loads a lot faster too:

Thursday

Now that I tested several sites and had a solid solution for generating multiple websites based on product-feeds, I had to take on the challenge of creating a generic homepage that would have to be as unique and specific as possible for each site to tell the user and the web-crawlers what the site is about. I looked at several other webshops and almost all of them had a header image or slider with multiple images to welcome the user and notify them of current promotions and new products. Because I don’t know these things just by retrieving the product-feed, I decided to “welcome” the user with a big font and a bit of text that would also be generated based on the feed and configuration. I had to add two more items to the config, one for “horloges” (watches) and another for “een horloge” (a watch) but this allows me to use one text with these two variables on all the websites, without having to copy-paste all the sentences and having to worry about typo’s. The homepage also shows 3 products of 4 different brands, I decided to sort the brands by alphabet and showing the last 4 in the list to make it seem more random. The list in the introduction text is also generated based on the feed, so I don’t need to know beforehand which brands there are in a feed. In the end, the homepage looks like this:

What’s next?

The concept has been proven and I can now publish the websites to test how well the sites are indexed in search engines. There are a lot of ideas and features to be added, like using multiple feeds on a single website, trying to merge those so I can offer price comparisons for the different websites, actually automating it with a cron-job and only re-generating files when there are different products and bots for setting up content and ads and posts for social media. I want the websites to become fully automated, from getting the feeds to noticing lowered prices and new products which will allow for a trigger to post this as an announcement to social media. The websites currently don’t have any sort of filtering or sorting aside from the categories, but because those pages are relying on their own JSON-feed it won’t be difficult to create filters in JavaScript. I also want to use this application to recreate the webshop I have with the dropshipper, but because there is no API I may have to manually create the product-feed myself. When I’ve added that webshop I want to build a pricing-bot that will compare prices of competing webshops with mine, maybe to guarantee the lowest price or to see what product I can put in the spotlight. When that’s all done I want to move on to fully-automated webshops with their own stock, in such a way that you only have to package, label and send the product.

But that will all have to wait for a while, because in the coming weeks I’ll be working hard on making major changes to Webtrimmer again.