Why have we become passionate about Augmented Reality at Metail?
It is because we believe a consumer’s first, digital experience with a fashion design should help consumers to quickly understand whether a particular design or brand is relevant to them. We are convinced that AR holds the key to realising this vision.
Alongside this, our experience with EcoShot has taught us that we can utilise and leverage the rich data contained in 3D CAD garments from the likes of Browzwear and CLO3D to help consumers. We therefore set about testing how to combine these two technologies into Augmented Reality try-on experiences for some of our customers.
Facemask AR try-on effect on Instagram
Our first project was especially topical given the pandemic that took over 2020–21: an augmented try-on experience for facemasks. Our starting point was using 3D-CAD-designed facemasks created in both Browzwear and CLO3D. For the Augmented Reality platform, we selected Facebook’s SparkAR. This is because we particularly wanted the AR experience to be available on the most popular social media platform for influencers and discovery of new apparel products: Instagram. The other motivation was to test if it was possible to combine our AR Try-On experience with the growing trend of social commerce and the Shopify x Instagram integration.
Learnings…
Whilst the initial reaction from the clients was delight, we then ran into a familiar foe — fabric representation. In particular, ensuring that fabrics’ colour in the Instagram AR effect matched that of the eCommerce studio photography. This is always a particularly challenging task given that it is difficult (and often impossible) to find out the lighting which the studio photography was taken under. In addition, another source of difficulty can be the 3D fabric scans. There is an art to taking good material scans and another art to setting them up correctly in your 3D tool of choice. Finding a way to smooth out this process is a problem we will need to solve.
In addition, we were disappointed to learn that creating a social commerce experience is also not a smooth experience. Despite the client’s website being built on Shopify and having created an Instagram Shop, there was no way for users to purchase the facemasks directly within the Instagram app. Instead, users were met with the blue “View on Website” button and the client could not identify how to activate Instagram Checkout, Facebook Pay or Shop Pay for users inside the app. Similarly, there did not appear to be a way to tag products in AR filters so that users could be taken directly to the mask’s product detail page within Instagram Shops. It seems like there is more work to be done for social commerce to live up to its promise outside of China. Hopefully the recent Shop Pay announcement to extend beyond Shopify merchants is a step in the right direction.
Outfit AR try-on lens for Snap
For our second project we really wanted to implement something more general for 3D-CAD-designed apparel from Browzwear this time. For the Augmented Reality platform, we selected Snap’s Len Studio. We would have probably been inclined to continue using Facebook’s Spark AR as Instagram is used by more people in the UK than Snapchat. However, Spark AR has a major omission which made it a no go for this project — no body tracking. This is slightly strange as we know Facebook have some of the best body tracking algorithms from research they’ve published and have even confirmed that these pose tracking algorithms run locally on their Portal video calling devices. But there was a silver lining to Facebook’s omission — it led us to trying Snap’s Lens Studio, which we’ve been thoroughly impressed by.
Learnings…
This was an R&D test and we did not spend time trying to productionise the process of going from a Browzwear 3D outfit to a Snap Lens. In spite of this, our developers were able to go from a Browzwear 3D CAD outfit to a Snap Lens in less than 30 minutes. This feels like a big win on the scalability side when taking into account:
- Browzwear customers have already created a large catalogue of 3D CAD garments
- Knowing that we will be able to automate the vast majority of the steps in the sub 30 minute process
On a less positive note, we learnt that hips can lie in AR! At least they did in an early iteration of our PUMA T7 outfit lens. I’m quite far off from being the curvy hourglass the AR lens suggests 😀
To solve our “big bum” problem, we had to tackle another familiar foe — differences in 3D file formats (a topic which David Gavilan has written a great blog post about). The solution was to add FBX model import into our internal avatar pipeline so that we could import the Lens Studio 4 template mesh and subsequently create a Browzwear VStitcher compatible avatar that we could simulate the outfits on. There was still one more bug we needed to solve whilst reposing during the skin transfer but once that was solved, my bum no longer looked big! The Snap lens publication process was then very smooth (although we did learn that you’re not allowed to include the word lens in the name of your lens!). You can try it for yourself using this Snapchat link or Snapcode…
The other major learnings were ideas we’ve had to improve the usefulness of this AR Try-On experience for consumers. A major drawback of our current lenses are the lack of accurate draping and realistic creasing. This has sparked ideas for a deep learning framework we could implement to improve the realism on this front. And we also currently lack a sense of absolute scale from Lens Studio. This will be an important requirement if we want to show correct sizing, which should be feasible from the outfit perspective given the rich data available to use within the Browzwear 3D CAD outfit.
Next steps
We’ll round off with how we started — our belief that a consumer’s first, digital experience with a fashion design should help them to quickly understand the relevance of the design and brand to them. These two small projects and our work in the eTryOn EU R&D consortium have indeed convinced us that AR holds the key to realising this vision. We’re looking forward to building more relevant, delightful and useful apparel experiences by introducing machine learning into the mix to ensure real time interactivity and design faithful AR representations. Plus we hope to see better social commerce and integration tools to make the experience far more seamless for consumers. But have we ended up feeling like this could be the future? The answer is a resounding yes and we are incredibly excited by it…and this is before smart glasses have even become a thing! We’re really enjoying seeing virtual try-on through a whole new lens 🔎 !
If you are involved in the apparel space and are excited by the prospect of AR letting people try on your clothes, do get in touch with us via the contact form on our Augmented Reality page.
